Synchronization in UBIFS (zero length files)
Hi Artem/Richard
I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @ http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:
"We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a big job. But feel free to do this :-)."
So two questions:
1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing something
2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
Thanks,
Tanya Brokhman
Signature · 2 lines
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Richard Weinberger
maintainerTanya,
Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
Quoted 11 lines
Hi Artem/Richard
I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:"We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a big job. But feel free to do this :-)."
So two questions:
1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing something
I don't think so.
Quoted 1 line
2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async. Having async IO for UBI is not trivial. Especially because of powercut safety.
Thanks,
//richard
Hi Richard,
Quoted 12 lines · Richard Weinberger
Tanya,
Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
Quoted 9 lines
Hi Artem/Richard
I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:"We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a big job. But feel free to do this :-)."
Quoted 8 lines
Quoted 2 lines
So two questions:
1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing somethingI don't think so.
Quoted 1 line
2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async.
Regarding EXT4. In Theodore Tso article about this (http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/) he mentioned 3 ext4 patches that are handling this issue but I can't find them by their git id:
"These three patches (with git id’s bf1b69c0, f32b730a, and 8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced."
Are you familiar with the patches in question?
Quoted 1 line
Having async IO for UBI is not trivial. Especially because of powercut safety.
Is it even doable? It seems to me that if we add async functionality to ubi it will become not power-cut tolerant, unless we implement a journal there (really don't want to do that....)
Another idea: from what I've read on the subject so far (this is new to me so be patient), zero-length files can result due to several use cases one of them is renaming files in the following scheme:
1. write to temp file A - written to cache
2. rename A to B - written directly to disk
3. eventually data is flashed to disk
If power cut occurs between #2 and #3 we end up with empty file B. But what if we call fdatasync on A as part of the rename operation?
More specifically: I'm looking @ fs/ubifs/dir.c-ubifs_rename(). What if we force sync of old_inode by:
1. err = old_inode->i_sb->s_op->write_inode(old_inode, NULL);
2. err = ubifs_sync_wbufs_by_inode(c, old_inode);
If I understand the code correctly it is basically the same as adding fsync() in userspace after #2 above.
What do you think?
Quoted 4 lines
Thanks,
//richard
Thanks,
Tanya Brokhman
Signature · 2 lines
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Richard Weinberger
maintainerTanya,
Am 18.12.2014 um 15:53 schrieb Tanya Brokhman:
Quoted 20 lines
Hi Richard,
Quoted 17 lines · Richard Weinberger
Tanya,
Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
Quoted 11 lines
Hi Artem/Richard
I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:"We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a big job. But feel free to do this :-)."
So two questions:
1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing somethingI don't think so.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async.
Regarding EXT4. In Theodore Tso article about this (http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/) he mentioned 3 ext4 patches that are handling this issue but I can't find them by their git id:
"These three patches (with git id’sbf1b69c0,f32b730a, and8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced." Are you familiar with the patches in question?
Nope.
Quoted 4 lines
Quoted 1 line
Having async IO for UBI is not trivial. Especially because of powercut safety.
Is it even doable? It seems to me that if we add async functionality to ubi it will become not power-cut tolerant, unless we implement a journal there (really don't want to do that....)
In software you can do everything but I fear it will be messy.
Quoted 11 lines
Another idea: from what I've read on the subject so far (this is new to me so be patient), zero-length files can result due to several use cases one of them is renaming files in the following scheme:
1. write to temp file A - written to cache
2. rename A to B - written directly to disk
3. eventually data is flashed to disk
If power cut occurs between #2 and #3 we end up with empty file B. But what if we call fdatasync on A as part of the rename operation?
More specifically: I'm looking @ fs/ubifs/dir.c-ubifs_rename(). What if we force sync ofold_inodeby:
1. err =old_inode->i_sb->s_op->write_inode(old_inode, NULL);
2. err =ubifs_sync_wbufs_by_inode(c, old_inode);
If I understand the code correctly it is basically the same as addingfsync()in userspace after #2 above.
Quoted 1 line
What do you think?
Let's wait what Artem says.
I bet here are more dragons. :)
Thanks,
//richard
Hi Artem,
Quoted 52 lines · Richard Weinberger
Tanya,
Am 18.12.2014 um 15:53 schrieb Tanya Brokhman:
Quoted 20 lines
Hi Richard,
Quoted 17 lines · Richard Weinberger
Tanya,
Am 18.12.2014 um 09:34 schrieb Tanya Brokhman:
Quoted 11 lines
Hi Artem/Richard
I've been looking into the "zero length files" issue in UBIFS and came across "Synchronization exceptions for buggy applications" @
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics. This section concludes with:"We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a big job. But feel free to do this :-)."
So two questions:
1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing somethingI don't think so.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
If we'd do it synchronous it would horrible slow down UBIFS. That's why ext4 is doing it async.
Regarding EXT4. In Theodore Tso article about this (http://thunk.org/tytso/blog/2009/03/12/delayed-allocation-and-the-zero-length-file-problem/) he mentioned 3 ext4 patches that are handling this issue but I can't find them by their git id:
"These three patches (with git id’sbf1b69c0,f32b730a, and8411e347) will cause a file to have any delayed allocation blocks to be allocated immediately when a file is replaced." Are you familiar with the patches in question?Nope.
Quoted 4 lines
Quoted 1 line
Having async IO for UBI is not trivial. Especially because of powercut safety.
Is it even doable? It seems to me that if we add async functionality to ubi it will become not power-cut tolerant, unless we implement a journal there (really don't want to do that....)
In software you can do everything but I fear it will be messy.
Quoted 11 lines
Another idea: from what I've read on the subject so far (this is new to me so be patient), zero-length files can result due to several use cases one of them is renaming files in the following scheme:
1. write to temp file A - written to cache
2. rename A to B - written directly to disk
3. eventually data is flashed to disk
If power cut occurs between #2 and #3 we end up with empty file B. But what if we call fdatasync on A as part of the rename operation?
More specifically: I'm looking @ fs/ubifs/dir.c-ubifs_rename(). What if we force sync ofold_inodeby:
1. err =old_inode->i_sb->s_op->write_inode(old_inode, NULL);
2. err =ubifs_sync_wbufs_by_inode(c, old_inode);
If I understand the code correctly it is basically the same as addingfsync()in userspace after #2 above.
Quoted 4 lines
Quoted 1 line
What do you think?
Let's wait what Artem says.
I bet here are more dragons. :)
Sorry to nag, this is a bit urgent for us. Could you please share your inputs on the above?
Thanks,
Tanya Brokhman
Signature · 2 lines
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Richard Weinberger
maintainerTanya,
Am 23.12.2014 um 09:26 schrieb Tanya Brokhman:
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
What do you think?
Let's wait what Artem says.
I bet here are more dragons. :)Sorry to nag, this is a bit urgent for us. Could you please share your inputs on the above?
we're all in X-mas mode so don't expect answers within the next 1-2 weeks from us. And to be honest, if this is an urgent issue for your employer which needs immediately action they have to hire one of us as consultant. This is not how community support works. Sorry for the hard words.
Thanks,
//richard
Quoted 8 lines · Richard Weinberger
Tanya,
Am 23.12.2014 um 09:26 schrieb Tanya Brokhman:
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
What do you think?
Let's wait what Artem says.
I bet here are more dragons. :)
Quoted 6 lines
Quoted 1 line
Sorry to nag, this is a bit urgent for us. Could you please share your inputs on the above?
we're all in X-mas mode so don't expect answers within the next 1-2 weeks from us. And to be honest, if this is an urgent issue for your employer which needs immediately action they have to hire one of us as consultant. This is not how community support works. Sorry for the hard words.
ok. sorry about that.
Marry Christmas all!
Thanks,
Tanya Brokhman
Signature · 2 lines
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Tanya,
Quoted 4 lines · Tanya Brokhman
So two questions:
1. was anything done in ubifs to handle file truncation and rename similar to ext4? Didn't find anything but afraid I might be missing something
Nope.
Quoted 2 lines
2. Not sure I understand why "implementing these features would require implementing asynchronous I/O in UBI". Could you please elaborate on this?
Well, the idea was to implement something like BIO for UBIFS. You submit your read or write requests, and go on doing your things. When your requests are done, you have your call-back function invoked. You may also go ahead and way for the completion of your request.
But we have the work queue API in Linux. Just create UBIFS work queue (this will essentially create a separate thread for you, AFAIK), submit write works there, let them write the closed-but-not-fsynced files there.
On unmount, flush, cancel all jobs and destroy the queue.
Just off-the top of my head.
Quoted 1 line · Artem Bityutskiy
Well, the idea was to implement something like BIO for UBIFS.
For UBI, of course. We even started doing this at some point, but then stopped. At that time I was confident we could do it.
The workqueue suggestion I gave may or may not work. It feels easier to implement and less intrusive. But you never know before you try.
Signature · 12 lines
Best Regards, Artem Bityutskiy --------------------------------------------------------------------- Intel Finland Oy Registered Address: PL 281, 00181 Helsinki Business Identity Code: 0357606 - 4 Domiciled in Helsinki This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Hi Artem
Quoted 9 lines · Artem
Quoted 1 line · Artem Bityutskiy
Well, the idea was to implement something like BIO for UBIFS.
For UBI, of course. We even started doing this at some point, but then stopped. At that time I was confident we could do it.
The workqueue suggestion I gave may or may not work. It feels easier to implement and less intrusive. But you never know before you try.
Thanks for the explanation!
This was a while back and the task already completed :) I found a long long discussion you had with the community back in 2009 when Ted Ts'o implemented the sync-on-rename/truncate hack for ext4. You were asking the developers opinion on whether same should be done for ubifs or the sync should remain the user responsibility. The conclusion was that it's user responsibility to sync before renaming so nothing was done for ubifs. Unfortunately our customer thought differently so I had to go through with adding this hack to ubifs. Not the most elegant solution to the zero-length-files problem, but the quickest and the safest for the moment.
Thanks,
Tanya Brokhman
Signature · 3 lines
Qualcomm Israel, on behalf of Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project