Skip to contentSkip to conversation

EXT3: problem with copy_from_user inside a transaction

RFC PATCH New Andrey Savochkin 3 participants 9 messages Sep 3, 2004 – Sep 7, 2004 VFS

lore
Share & tools
lore
#0 lore raw

Hi Andrew,

filemap_copy_from_user() between prepare_write() and commit_write() appears to be a problem for ext3.

prepare_write() starts a transaction, and if filemap_copy_from_user() causes a page fault, we'll have
- order violation with mmap_sem taken inside a transaction (possible deadlocks),
- __GFP_FS memory allocation with all re-entrancy problems (e.g., current->journal_info corruption).

Am I missing something?

If this observation is correct, the possible solution is to call get_user_pages() or somehow pin the user pages before prepare_write(), although it will hurt performance...

Andrey

#1 lore raw
Quoted 12 lines · Andrey Savochkin

Hi Andrew,

filemap_copy_from_user() between prepare_write() and commit_write() appears to be a problem for ext3.

prepare_write() starts a transaction, and if filemap_copy_from_user() causes a page fault, we'll have
- order violation with mmap_sem taken inside a transaction (possible deadlocks),
- __GFP_FS memory allocation with all re-entrancy problems (e.g., current->journal_info corruption).

Quoted 5 lines

Am I missing something?

If this observation is correct, the possible solution is to call get_user_pages() or somehow pin the user pages before prepare_write(), although it will hurt performance...

yes, Chris is working on it for a few months.

just for the mmap_sem the easiest fix I proposed him, was to take the mmap_sem in read mode before starting the transaction in prepare_write, then the mmap_sem has to become recursive but that's trivial (my 2.4 rw semaphores are much simpler, they don't crash with million of waiters, and they can support recursion).

however it seems the mmap_sem is not the only issue, he found another issue with the page lock, maybe Chris you want to elaborate (the above deadlock is absolutely clear, the one with the page lock less, btw, I don't care much with reading the same page from disk that we're writing to in the write syscall, that's a case we may just want to forbidden since it makes no sense and currently it's racey even in 2.4, no pin happens, but the prefaulting hides it).

We also hidden the above deadlock with prefaulting (but it's far from fixed), prefaulting is a good idea anyways.

#2 lore raw
Quoted 6 lines · Andrea Arcangeli
Quoted 5 lines · Andrey Savochkin

Hi Andrew,

filemap_copy_from_user() between prepare_write() and commit_write() appears to be a problem for ext3.

And reiserv3, and maybe the other journaled filesystems.

Quoted 2 lines

yes, Chris is working on it for a few months.

Working is a generous term, I've somewhat been waiting for a better solution to pop into my head. In the end, I think all we can do is not allow filesystems to take locks (or implicit locks like starting a transaction) inside the prepare_write call.

This would mean that all the work is done during the commit_write stage. The trick is that we would have to handle -ENOSPC since we might not know we've run out of room until after the data has been copied from userland.

prepare_write could reserve blocks, which brings us half way to a generic delayed allocation layer. But for a quick and dirty start, doing it all in commit_write should work.

-chris

#3 lore raw
Quoted 2 lines · Chris Mason

prepare_write could reserve blocks, which brings us half way to a generic delayed allocation layer. [..]

sounds good to me!

#4 lore raw
In reply to Chris Mason
Quoted 15 lines · Chris Mason
Quoted 5 lines · Andrea Arcangeli
Quoted 4 lines · Andrey Savochkin

filemap_copy_from_user() between prepare_write() and commit_write() appears to be a problem for ext3.

And reiserv3, and maybe the other journaled filesystems.

Quoted 2 lines

yes, Chris is working on it for a few months.

Working is a generous term, I've somewhat been waiting for a better solution to pop into my head. In the end, I think all we can do is not allow filesystems to take locks (or implicit locks like starting a transaction) inside the prepare_write call.

Quoted 4 lines

This would mean that all the work is done during the commit_write stage. The trick is that we would have to handle -ENOSPC since we might not know we've run out of room until after the data has been copied from userland.

What is the problem -ENOSPC?
Do you think about the problem of the page existing before this write, it's content overwritten, but the filesystem being unable to commit that write because it needs more space?

Andrey

#5 lore raw
Quoted 9 lines · Andrey Savochkin
Quoted 4 lines

This would mean that all the work is done during the commit_write stage. The trick is that we would have to handle -ENOSPC since we might not know we've run out of room until after the data has been copied from userland.

What is the problem -ENOSPC?
Do you think about the problem of the page existing before this write, it's content overwritten, but the filesystem being unable to commit that write because it needs more space?

Exactly. In this case, we've effectively corrupted the page cache. We've copied data in that isn't (and never will be) reflected on disk. It isn't a horribly difficult case, we just need to overwrite the data with zeros, making sure to only overwrite the data corresponding to the -ENOSPC error.

-chris

#6 lore raw
Quoted 12 lines · Chris Mason
Quoted 9 lines · Andrey Savochkin
Quoted 4 lines

This would mean that all the work is done during the commit_write stage. The trick is that we would have to handle -ENOSPC since we might not know we've run out of room until after the data has been copied from userland.

What is the problem -ENOSPC?
Do you think about the problem of the page existing before this write, it's content overwritten, but the filesystem being unable to commit that write because it needs more space?

Quoted 5 lines

Exactly. In this case, we've effectively corrupted the page cache. We've copied data in that isn't (and never will be) reflected on disk. It isn't a horribly difficult case, we just need to overwrite the data with zeros, making sure to only overwrite the data corresponding to the -ENOSPC error.

Zeroing not mapped buffers in case of error is not difficult, indeed.

I was speaking about the following case:
- one page of a file is dirtied through a userspace mapping,
- the page doesn't have disk mapping yet (a hole),
- someone issues write() to this page,
- the space allocation fails in commit_write(), when the page content has already been overwritten with the new data.

Any holes in this scenario? :)

How to handle -ENOSPC in this case?

Andrey

#7 lore raw
Quoted 13 lines · Andrey Savochkin
Quoted 11 lines · Chris Mason
Quoted 4 lines · Andrey Savochkin

What is the problem -ENOSPC?
Do you think about the problem of the page existing before this write, it's content overwritten, but the filesystem being unable to commit that write because it needs more space?

Exactly. In this case, we've effectively corrupted the page cache. We've copied data in that isn't (and never will be) reflected on disk. It isn't a horribly difficult case, we just need to overwrite the data with zeros, making sure to only overwrite the data corresponding to the -ENOSPC error.

Quoted 8 lines

Zeroing not mapped buffers in case of error is not difficult, indeed.

I was speaking about the following case:
- one page of a file is dirtied through a userspace mapping,
- the page doesn't have disk mapping yet (a hole),
- someone issues write() to this page,
- the space allocation fails in commit_write(), when the page content has already been overwritten with the new data.

After some thought, we can check if the page is not completely mapped to disk and there is a possibility that it's referenced from pte's.
In this case we can "commit" the old content in prepare_write(), allocating space and instantiating holes...

Any better ideas?

Andrey

#8 lore raw

RFC PATCH 1/1 EXT3: problem with copy_from_user inside a transaction

In reply to Chris Mason
Quoted 16 lines · Chris Mason
Quoted 6 lines · Andrea Arcangeli
Quoted 5 lines · Andrey Savochkin

Hi Andrew,

filemap_copy_from_user() between prepare_write() and commit_write() appears to be a problem for ext3.

And reiserv3, and maybe the other journaled filesystems.

Quoted 2 lines

yes, Chris is working on it for a few months.

Working is a generous term, I've somewhat been waiting for a better solution to pop into my head. In the end, I think all we can do is not allow filesystems to take locks (or implicit locks like starting a transaction) inside the prepare_write call.

Quoted 8 lines

This would mean that all the work is done during the commit_write stage. The trick is that we would have to handle -ENOSPC since we might not know we've run out of room until after the data has been copied from userland.

prepare_write could reserve blocks, which brings us half way to a generic delayed allocation layer. But for a quick and dirty start, doing it all in commit_write should work.

Ok, so here is the patch moving journal_start() together with space allocation from ext3_prepare_write() to commit_write().

ENOSPC is handled in ext3_map_write() called from commit_write(). In ENOSPC or other error case, the data copied from the userspace is zeroed, but only if the buffers and the whole page are not up to date. Answering my previous questions, if
- the inode page is modified through mmap and then by write,
- the file has holes and
- there is no space on disk,
the page cache will have the new content (provided by write) not written to disk. Similarly to modifications through pure mmap.

It's interesting that block_prepare_write() zeroes the page content in case of error even if the page has been modified through mmap()...

Comments?

Andrey

===== fs/buffer.c 1.255 vs edited =====

fs/buffer.c +3 -2
@@ -2025,8 +2025,9 @@ static int __block_prepare_write(struct 
 				goto out;
 			if (buffer_new(bh)) {
 				clear_buffer_new(bh);
-				unmap_underlying_metadata(bh->b_bdev,
-							bh->b_blocknr);
+				if (buffer_mapped(bh))
+					unmap_underlying_metadata(bh->b_bdev,
+								bh->b_blocknr);
 				if (PageUptodate(page)) {
 					set_buffer_uptodate(bh);
 					continue;

===== fs/ext3/inode.c 1.101 vs edited =====

fs/ext3/inode.c +138 -17

220 lines not on this page · raw message

===== fs/jbd/transaction.c 1.87 vs edited =====

fs/jbd/transaction.c +1 -0
@@ -1870,6 +1870,7 @@ zap_buffer_unlocked:
 	clear_buffer_mapped(bh);
 	clear_buffer_req(bh);
 	clear_buffer_new(bh);
+	clear_buffer_delay(bh);
 	bh->b_bdev = NULL;
 	return may_free;
 }