Input/output error on newly created file
Hello,
I just created an innocent 10 MB on a btrfs file system, yet my attempt to read it a few seconds later (and ever since), just gives:
$ ls -l in-progress/mysterious-io-error -rw-rw-r-- 1 nikratio nikratio 10485760 May 8 14:41 in-progress/mysterious-io-error $ cat in-progress/mysterious-io-error cat: in-progress/mysterious-io-error: Input/output error
I was hoping the kernel might tell me a little more about this, but:
$ dmesg | tail -3 [ 48.982851] wlan0: RX AssocResp from 00:1f:33:be:9a:c4 (capab=0x411 status=0 aid=3) [ 48.983757] wlan0: associated [ 2770.464505] kvm: zapping shadow pages for mmio generation wraparound
How can I find out the reason for the error? Is it a bug in BTRFS, or somewhere else in the kernel ? Is my hardware broken?
Kernel is Debian's 4.5.0-0.bpo.1-amd64. This is a btrfs file system on a LVM LV backed by a bcache array with two backing devices and one caching device.
Best,
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 9 lines · Nikolaus Rath
Hello,
I just created an innocent 10 MB on a btrfs file system, yet my attempt to read it a few seconds later (and ever since), just gives:
$ ls -l in-progress/mysterious-io-error -rw-rw-r-- 1 nikratio nikratio 10485760 May 8 14:41 in-progress/mysterious-io-error $ cat in-progress/mysterious-io-error cat: in-progress/mysterious-io-error: Input/output error
If you unmount and mount again the filesystem, does it happen again? How did you create the file? Does fsck reports any issues?
Quoted 29 lines
I was hoping the kernel might tell me a little more about this, but:
$ dmesg | tail -3 [ 48.982851] wlan0: RX AssocResp from 00:1f:33:be:9a:c4 (capab=0x411 status=0 aid=3) [ 48.983757] wlan0: associated [ 2770.464505] kvm: zapping shadow pages for mmio generation wraparoundHow can I find out the reason for the error? Is it a bug in BTRFS, or somewhere else in the kernel ? Is my hardware broken?
Kernel is Debian's 4.5.0-0.bpo.1-amd64. This is a btrfs file system on a LVM LV backed by a bcache array with two backing devices and one caching device.
Best,
-Nikolaus--
GPG encrypted emails preferred. Key id:0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F»Time flies like an arrow, fruit flies like a Banana.«
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 5 lines
Filipe David Manana, "Reasonable men adapt themselves to the world. Unreasonable men adapt the world to themselves. That's why all progress depends on unreasonable men."
Quoted 6 lines · Filipe Manana
Quoted 5 lines · Nikolaus Rath
Hello,
I just created an innocent 10 MB on a btrfs file system, yet my attempt to read it a few seconds later (and ever since), just gives:
Quoted 6 lines
Quoted 4 lines
$ ls -l in-progress/mysterious-io-error -rw-rw-r-- 1 nikratio nikratio 10485760 May 8 14:41 in-progress/mysterious-io-error $ cat in-progress/mysterious-io-error cat: in-progress/mysterious-io-error: Input/output errorIf you unmount and mount again the filesystem, does it happen again?
After rebooting, the previously unaccessible file can now be read. But I cannot tell if it contains the right data.
However, I just encountered the same problem with another, freshly created file.
Quoted 1 line
How did you create the file?
In Python 3. The equivalent code is more or less:
with open('file.dat', 'wb+') as fh:
for buf in generate_data():
fh.write(buf) # bufsize is about 128 kB
However, I should note that there is a lot of activity in this file system (it contains my home directory), so the above alone will probably not reproduce the problem.
That said, so far both the problematic files were created by the same application (S3QL, of which luckily I am also the maintainer).
Quoted 1 line
Does fsck reports any issues?
Do you mean btrfsck? It actually has a lot to say:
checking extents
checking free space cache
checking fs roots
root 5 inode 3149867 errors 400, nbytes wrong root 5 inode 3150237 errors 400, nbytes wrong root 5 inode 3150238 errors 400, nbytes wrong root 5 inode 3150242 errors 400, nbytes wrong root 5 inode 3150260 errors 400, nbytes wrong [...]
root 5 inode 15595011 errors 400, nbytes wrong root 5 inode 15595016 errors 400, nbytes wrong Checking filesystem on /dev/mapper/vg0-nikratio_crypt
UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38 found 263648960636 bytes used err is 1
total csum bytes: 395314372
total tree bytes: 908644352
total fs tree bytes: 352735232
total extent tree bytes: 95039488
btree space waste bytes: 156301160
file data blocks allocated: 675209801728
referenced 410351722496
Btrfs v3.17
However, the inode of the problematic file (16186241) is not mentioned. But I guess this is not surprising, because also for this file, I can read the contents after remounting.
Best,
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«*ping*
Anyone any idea?
Best,
-Nikolaus
Quoted 82 lines · Nikolaus Rath
Quoted 6 lines · Filipe Manana
Quoted 5 lines · Nikolaus Rath
Hello,
I just created an innocent 10 MB on a btrfs file system, yet my attempt to read it a few seconds later (and ever since), just gives:
Quoted 6 lines
Quoted 4 lines
$ ls -l in-progress/mysterious-io-error -rw-rw-r-- 1 nikratio nikratio 10485760 May 8 14:41 in-progress/mysterious-io-error $ cat in-progress/mysterious-io-error cat: in-progress/mysterious-io-error: Input/output errorIf you unmount and mount again the filesystem, does it happen again?
After rebooting, the previously unaccessible file can now be read. But I cannot tell if it contains the right data.
However, I just encountered the same problem with another, freshly created file.
Quoted 1 line
How did you create the file?
In Python 3. The equivalent code is more or less:
with
open('file.dat', 'wb+')as fh:
for buf ingenerate_data():
fh.write(buf)# bufsize is about 128 kBHowever, I should note that there is a lot of activity in this file system (it contains my home directory), so the above alone will probably not reproduce the problem.
That said, so far both the problematic files were created by the same application (S3QL, of which luckily I am also the maintainer).
Quoted 1 line
Does fsck reports any issues?
Do you mean btrfsck? It actually has a lot to say:
checking extents
checking free space cache
checking fs roots
root 5 inode 3149867 errors 400, nbytes wrong root 5 inode 3150237 errors 400, nbytes wrong root 5 inode 3150238 errors 400, nbytes wrong root 5 inode 3150242 errors 400, nbytes wrong root 5 inode 3150260 errors 400, nbytes wrong [...]
root 5 inode 15595011 errors 400, nbytes wrong root 5 inode 15595016 errors 400, nbytes wrong Checking filesystem on/dev/mapper/vg0-nikratio_crypt
UUID: 8742472d-a9b0-4ab6-b67a-5d21f14f7a38 found 263648960636 bytes used err is 1
total csum bytes: 395314372
total tree bytes: 908644352
total fs tree bytes: 352735232
total extent tree bytes: 95039488
btree space waste bytes: 156301160
file data blocks allocated: 675209801728
referenced 410351722496
Btrfs v3.17However, the inode of the problematic file (16186241) is not mentioned. But I guess this is not surprising, because also for this file, I can read the contents after remounting.
Best,
-NikolausSignature · 8 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 3 lines · El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió:
*ping*
Anyone any idea?
All I can say is that I've had the same problem in the past. In my case, the problematic files where active torrents. The interesting thing is that I was able to read them correctly up to a point, then I would get the same error as you. No messages in dmesg. The amount of data I was able to read from them was not random, it was something multiple of 4K. After reboot the problems went away and I wasn't able to reproduce it.
There has been reports of similiar issues in the past:
http://www.spinics.net/lists/linux-btrfs/msg52371.html
Quoted 13 lines · Diego Calleja
Quoted 3 lines · El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió:
*ping*
Anyone any idea?
All I can say is that I've had the same problem in the past. In my case, the problematic files where active torrents. The interesting thing is that I was able to read them correctly up to a point, then I would get the same error as you. No messages in dmesg. The amount of data I was able to read from them was not random, it was something multiple of 4K. After reboot the problems went away and I wasn't able to reproduce it.
Quoted 2 lines
There has been reports of similiar issues in the past:
http://www.spinics.net/lists/linux-btrfs/msg52371.html
Thanks for the pointer. So according to these reports, the IO errors happen with:
1. dm-crypt, LVM and mount options rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
2. dm-crypt, no LVM, and mount options
noatime,compress,nossd
I can add the following data point:
3. dm-crypt on LVM, and mount options
relatime,compress=lzo
(Just to preserve this for the future)
Best,
-Nikolaus
Signature · 4 lines
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«2016-05-12 19:41 keltezéssel, Nikolaus Rath írta:
Quoted 24 lines · Diego Calleja
Quoted 12 lines · El jueves, 12 de mayo de 2016 8:46:00 (CEST) Nikolaus Rath escribió:
Quoted 3 lines
*ping*
Anyone any idea?
All I can say is that I've had the same problem in the past. In my case, the problematic files where active torrents. The interesting thing is that I was able to read them correctly up to a point, then I would get the same error as you. No messages in dmesg. The amount of data I was able to read from them was not random, it was something multiple of 4K. After reboot the problems went away and I wasn't able to reproduce it.
Quoted 2 lines
There has been reports of similiar issues in the past:
http://www.spinics.net/lists/linux-btrfs/msg52371.htmlThanks for the pointer. So according to these reports, the IO errors happen with:
1. dm-crypt, LVM and mount options rw,noatime,compress=lzo,ssd,discard,space_cache,autodefrag,inode_cache
2. dm-crypt, no LVM, and mount options
noatime,compress,nossd
Quoted 7 lines
I can add the following data point:
3. dm-crypt on LVM, and mount options
relatime,compress=lzo(Just to preserve this for the future)
I don't want to repeat my e-mails in the past, but I think I have a similar issue. In short: xen virtual machine. The files that rarely become unreadable (I/O error but no error in dmesg or anywhere) are mysql MyIsam database files, and they are always small. Like 16kbyte for example, or smaller. Sometimes dropping the fs cache fixes the problem, sometimes not. Umount / mount always fixes the problem. Scrub says the filesystem is OK (when the file is unreadable).
I experienced similar problem with log files (smtp or apache log files), but it is rare. It happens 1-2 times in a month on a heavy loaded mail/web server. (the log file io error is not a real problem for me, mysql is.). The problem present from 3.18 to 4.5.
I started to migrate mysql table files to innodb, but i think this is only a workaround.
László Szalma
Quoted 4 lines
Best,
-Nikolaus
Szalma László posted on Thu, 12 May 2016 20:28:24 +0200 as excerpted:
Quoted 5 lines
The files that rarely become unreadable (I/O error but no error in dmesg or anywhere) are mysql MyIsam database files, and they are always small. Like 16kbyte for example, or smaller. Sometimes dropping the fs cache fixes the problem, sometimes not. Umount / mount always fixes the problem. Scrub says the filesystem is OK (when the file is unreadable).
Is it possible the files are always under 4 KiB?
While there's a few variables as to max size, with 4 KiB being the practical max, btrfs will inline really small files into their metadata node instead of writing them out as 4 KiB data block extents. Since that's obviously a different code-path, if it's only 4 KiB and smaller files, it's possible there's a race in that code-path that when lost, results in the file "disappearing" without a dmesg trace, only to reappear after reboot.
Actually, now that I think about it, if the files are all OVER say 2 KiB but still quite small, say under 16 KiB, and if they had just recently grown, it's possible that the race is in the transition from the inline metadata state to the multiples of 4 KiB block data extent state.
And if all the files had just shrunk, say from compaction (if done in- place, not with a copy and rename), perhaps it's the reverse, the transition from written data blocks to inline metadata state.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Am Friday 13 May 2016
schrieb Duncan <[email protected]>
Quoted 8 lines
Szalma László posted on Thu, 12 May 2016 20:28:24 +0200 as excerpted:
Quoted 5 lines
The files that rarely become unreadable (I/O error but no error in dmesg or anywhere) are mysql MyIsam database files, and they are always small. Like 16kbyte for example, or smaller. Sometimes dropping the fs cache fixes the problem, sometimes not. Umount / mount always fixes the problem. Scrub says the filesystem is OK (when the file is unreadable).
Is it possible the files are always under 4 KiB?
For the record, I was seeing a similar error with dovecot *.index.log files (see the ML thread started by Szalma László) . In my case they are *not* all under 4 KiB. Looking at some of the affected files, one of them is 25K, and another is 6.6K. However, perhaps they compress to under 4K? But compressing the 25K one with lzop only goes down to 5.6K with -9 :-/ .
Quoted 8 lines
While there's a few variables as to max size, with 4 KiB being the practical max, btrfs will inline really small files into their metadata node instead of writing them out as 4 KiB data block extents. Since that's obviously a different code-path, if it's only 4 KiB and smaller files, it's possible there's a race in that code-path that when lost, results in the file "disappearing" without a dmesg trace, only to reappear after reboot.
Quoted 8 lines
Actually, now that I think about it, if the files are all OVER say 2 KiB but still quite small, say under 16 KiB, and if they had just recently grown, it's possible that the race is in the transition from the inline metadata state to the multiples of 4 KiB block data extent state.
And if all the files had just shrunk, say from compaction (if done in- place, not with a copy and rename), perhaps it's the reverse, the transition from written data blocks to inline metadata state.
I'm glad somebody is (publicly) thinking about this :-) !
Greetings
Signature · 4 lines
Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup