Skip to contentSkip to conversation

Q about pagecache data never written to disk

DISCUSSION Andrey Savochkin 5 participants 17 messages Sep 5, 2004 – Sep 9, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Let's suppose an mmap'ed (SHARED, RW) file has a hole.
AFAICS, we allow to dirty the file pages without allocating the space for the hole - filemap_nopage just "reads" the page filling it with zeroes, and nothing is done about the on-disk data until writepage.

So, if the page can't be written to disk (no space), the dirty data just stays in the pagecache. The data can be read or seen via mmap, but it isn't and never be on disk. The pagecache stays unsynchronized with the on-disk content forever.

Is it the intended behavior?
Shouldn't we call the filesystem to fill the hole at the moment of the first write access?

Andrey

#1 lore raw
Quoted 11 lines · Andrey Savochkin

Let's suppose an mmap'ed (SHARED, RW) file has a hole.
AFAICS, we allow to dirty the file pages without allocating the space for the hole - filemap_nopage just "reads" the page filling it with zeroes, and nothing is done about the on-disk data until writepage.
So, if the page can't be written to disk (no space), the dirty data just stays in the pagecache. The data can be read or seen via mmap, but it isn't and never be on disk. The pagecache stays unsynchronized with the on-disk content forever.
Is it the intended behavior?
Shouldn't we call the filesystem to fill the hole at the moment of the first write access?

We would have to trap the first write access for that. What we do at the moment is lazily collecting the results of write accesses a.k.a. dirty bits from the pte data structures only under memory pressure. i.e. mmap() IO is permabust, and fixing it is permavetoed. At the moment the only protection faults the kernel understands are those meant for copy-on-write; trapping these accesses involves understanding that a protection fault may occur for other reasons.

So, in do_no_page() we now have:

   1558      * Note that if write_access is true, we either now have
   1559      * an exclusive copy of the page, or this is a shared mapping,
   1560      * so we can make it writable and dirty to avoid having to
   1561      * handle that later.
   1562      */
   1563     /* Only go through if we didn't race with anybody else... */
   1564     if (pte_none(*page_table)) {
   1565         if (!PageReserved(new_page))
   1566             ++mm->rss;
   1567         flush_icache_page(vma, new_page);
   1568         entry = mk_pte(new_page, vma->vm_page_prot);
   1569         if (write_access)

1570 entry = maybe_mkwrite(pte_mkdirty(entry), vma);
1571 set_pte(page_table, entry);

Here we would have to rearrange the pte setup, something vaguely like, but not precisely like, the following snippet.

The basic idea is that you arrange for the event to occur in do_no_page() when a read fault is taken on the thing, and then handle it later in do_no_page(), or otherwise process it immediately if you know it's happening right in do_no_page(). It probably makes sense to do set_page_dirty() and other things around this kind of situation as well, and the calling convention in this example may not be ideal. At any rate, if the space reservation for the page may block, you'll have to spin_unlock(&vma->vm_mm->page_table_lock) and also have to pte_unmap(page_table) in enough_space() in this arrangement.

Of course, you won't be able to use this out of the box; you'll have to implement enough_space() and possibly even add a new filesystem method for enough_space() to call to do its ENOSPC detection. This may also not interoperate particularly well with architecture support code for non-i386 architectures. I see low enough odds of this kind of affair getting merged that I don't really see a point in going through with much of this myself, though if you yourself have a need, maybe this tells you something useful enough for you to carry out the rest.

There was some kind of talk of an alternative to be carried out at mmap() -time, but as of yet there's been no coherent explanation of how it's possible for such half measures to cope with the realities of block indexing metadata or space consumers competing with mmap().

-- wli

mm/memory.c +27 -3
@@ -1056,6 +1056,19 @@ static int do_wp_page(struct mm_struct *
 	unsigned long pfn = pte_pfn(pte);
 	pte_t entry;
 
+	if (vma->vm_flags & VM_WRITE) {
+		int ret;
+		if (enough_space(vma, address, pte, &page_table)) {
+			ret = VM_FAULT_MINOR;
+			pte = pte_mkwrite(pte_mkyoung(pte_mkdirty(pte))));
+			set_pte(page_table, pte);
+		} else
+			ret = VM_FAULT_SIGBUS;
+		pte_unmap(page_table);
+		spin_unlock(&mm->page_table_lock);
+		return ret;
+	}
+
 	if (unlikely(!pfn_valid(pfn))) {
 		/*
 		 * This should really halt the system so it can be debugged or
@@ -1562,13 +1575,24 @@ do_no_page(struct mm_struct *
 	 */
 	/* Only go through if we didn't race with anybody else... */
 	if (pte_none(*page_table)) {
-		if (!PageReserved(new_page))
-			++mm->rss;
 		flush_icache_page(vma, new_page);
 		entry = mk_pte(new_page, vma->vm_page_prot);
 		if (write_access)
 			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
-		set_pte(page_table, entry);
+		else
+			entry = pte_wrprotect(pte);
+		if (!write_access ||
+				enough_space(vma, address, pte, &page_table))
+			set_pte(page_table, entry);
+		else {
+			spin_unlock(&mm->page_table_lock);
+			pte_unmap(page_table);
+			ret = VM_FAULT_SIGBUS;
+			page_cache_release(new_page);
+			goto out;
+		}
+		if (!PageReserved(new_page))
+			++mm->rss;
 		if (anon) {
 			lru_cache_add_active(new_page);
 			page_add_anon_rmap(new_page, vma, address);
#2 lore raw
In reply to Andrey Savochkin
Quoted 10 lines · Andrey Savochkin

Let's suppose an mmap'ed (SHARED, RW) file has a hole.
AFAICS, we allow to dirty the file pages without allocating the space for the hole - filemap_nopage just "reads" the page filling it with zeroes, and nothing is done about the on-disk data until writepage.

So, if the page can't be written to disk (no space), the dirty data just stays in the pagecache. The data can be read or seen via mmap, but it isn't and never be on disk. The pagecache stays unsynchronized with the on-disk content forever.

The kernel will make one attampt to write the data to disk. If that write hits ENOSPC, the page is not redirtied (ie: the data can be lost).

When that write hits ENOSPC an error flag is set in the address_space and that will be returned from a subsequent msync(). The application will then need to do something about it.

If your application doesn't msync() the memory then it doesn't care about its data anyway. If your application _does_ msync the pages then we reliably report errors.

Quoted 3 lines

Is it the intended behavior?
Shouldn't we call the filesystem to fill the hole at the moment of the first write access?

That would be a retrograde step - it would be nice to move in the other
direction: perform disk allocation at writeback time rather than at write() time, even for regular write() data. To do that we (probably) need space reservation APIs. And yes, we perhaps could reserve space in the filesystem when that page is first written to.

But then what would we do if there's no space? SIGBUS? SIGSEGV? Inappropriate. SIGENOSPC?

#3 lore raw

Hi Andrew,

Quoted 15 lines · Andrew Morton
Quoted 10 lines · Andrey Savochkin

Let's suppose an mmap'ed (SHARED, RW) file has a hole.
AFAICS, we allow to dirty the file pages without allocating the space for the hole - filemap_nopage just "reads" the page filling it with zeroes, and nothing is done about the on-disk data until writepage.

So, if the page can't be written to disk (no space), the dirty data just stays in the pagecache. The data can be read or seen via mmap, but it isn't and never be on disk. The pagecache stays unsynchronized with the on-disk content forever.

The kernel will make one attampt to write the data to disk. If that write hits ENOSPC, the page is not redirtied (ie: the data can be lost).

Quoted 7 lines

When that write hits ENOSPC an error flag is set in the address_space and that will be returned from a subsequent msync(). The application will then need to do something about it.

If your application doesn't msync() the memory then it doesn't care about its data anyway. If your application _does_ msync the pages then we reliably report errors.

This question came to my mind when I was thinking about journal_start in ext3_prepare_write and copy_from_user issue...
Did you follow that discussion?

In the considered scenario not only the application is not guaranteed anything till msync(), but all other programs doing regular read() may also be fooled about the file content, and this idea surprised me. On the other hand, after a write() other programs also see the new content without a guarantee that this content corresponds with what is on the disk...

Quoted 5 lines
Quoted 4 lines

Is it the intended behavior?
Shouldn't we call the filesystem to fill the hole at the moment of the first write access?

Quoted 8 lines

That would be a retrograde step - it would be nice to move in the other
direction: perform disk allocation at writeback time rather than at write() time, even for regular write() data. To do that we (probably) need space reservation APIs. And yes, we perhaps could reserve space in the filesystem when that page is first written to.

But then what would we do if there's no space? SIGBUS? SIGSEGV? Inappropriate. SIGENOSPC?

Should the space be allocated on close()?
Who will get the signal if nobody accesses the file anymore? I'm also thinking about various shell scripts with redirects to files...

Andrey

#4 lore raw
Quoted 23 lines · Andrey Savochkin

Hi Andrew,

Quoted 19 lines · Andrew Morton
Quoted 10 lines · Andrey Savochkin

Let's suppose an mmap'ed (SHARED, RW) file has a hole.
AFAICS, we allow to dirty the file pages without allocating the space for the hole - filemap_nopage just "reads" the page filling it with zeroes, and nothing is done about the on-disk data until writepage.

So, if the page can't be written to disk (no space), the dirty data just stays in the pagecache. The data can be read or seen via mmap, but it isn't and never be on disk. The pagecache stays unsynchronized with the on-disk content forever.

The kernel will make one attampt to write the data to disk. If that write hits ENOSPC, the page is not redirtied (ie: the data can be lost).

When that write hits ENOSPC an error flag is set in the address_space and that will be returned from a subsequent msync(). The application will then need to do something about it.

Quoted 7 lines
Quoted 3 lines

If your application doesn't msync() the memory then it doesn't care about its data anyway. If your application _does_ msync the pages then we reliably report errors.

This question came to my mind when I was thinking about journal_start in ext3_prepare_write and copy_from_user issue...
Did you follow that discussion?

Yup. Chris and I have been admiring the problem for a few months now.

Quoted 5 lines

In the considered scenario not only the application is not guaranteed anything till msync(), but all other programs doing regular read() may also be fooled about the file content, and this idea surprised me. On the other hand, after a write() other programs also see the new content without a guarantee that this content corresponds with what is on the disk...

No, read() will see the modified pagecache data immediately, apart from CPU cache coherency effects.

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

Is it the intended behavior?
Shouldn't we call the filesystem to fill the hole at the moment of the first write access?

That would be a retrograde step - it would be nice to move in the other
direction: perform disk allocation at writeback time rather than at write() time, even for regular write() data. To do that we (probably) need space reservation APIs. And yes, we perhaps could reserve space in the filesystem when that page is first written to.

Quoted 4 lines
Quoted 2 lines

But then what would we do if there's no space? SIGBUS? SIGSEGV? Inappropriate. SIGENOSPC?

Should the space be allocated on close()?

What effect are you trying to achieve?

Quoted 1 line

Who will get the signal if nobody accesses the file anymore?

Nobody. That's the point. Plus there _is_ no signal defined for this. Neither in Linux nor in POSIX.

Quoted 1 line

I'm also thinking about various shell scripts with redirects to files...

? I doubt that they're writing files via MAP_SHARED.

#5 lore raw
Quoted 8 lines · Andrew Morton
Quoted 7 lines · Andrey Savochkin
Quoted 6 lines · Andrew Morton

That would be a retrograde step - it would be nice to move in the other
direction: perform disk allocation at writeback time rather than at write() time, even for regular write() data. To do that we (probably) need space reservation APIs. And yes, we perhaps could reserve space in the filesystem when that page is first written to.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

But then what would we do if there's no space? SIGBUS? SIGSEGV? Inappropriate. SIGENOSPC?

Should the space be allocated on close()?

What effect are you trying to achieve?

Sending a signal while there is still a process...

Quoted 8 lines
Quoted 1 line

Who will get the signal if nobody accesses the file anymore?

Nobody. That's the point. Plus there _is_ no signal defined for this. Neither in Linux nor in POSIX.

Quoted 1 line

I'm also thinking about various shell scripts with redirects to files...

? I doubt that they're writing files via MAP_SHARED.

I was deliberating on your idea about delayed allocation for regular write()s also...

#6 lore raw
In reply to Andrew Morton

Hi!

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines

The kernel will make one attampt to write the data to disk. If that write hits ENOSPC, the page is not redirtied (ie: the data can be lost).

When that write hits ENOSPC an error flag is set in the address_space and that will be returned from a subsequent msync(). The application will then need to do something about it.

Quoted 7 lines
Quoted 3 lines

If your application doesn't msync() the memory then it doesn't care about its data anyway. If your application _does_ msync the pages then we reliably report errors.

This question came to my mind when I was thinking about journal_start in ext3_prepare_write and copy_from_user issue...
Did you follow that discussion?

Yup. Chris and I have been admiring the problem for a few months now.

Quoted 8 lines
Quoted 5 lines

In the considered scenario not only the application is not guaranteed anything till msync(), but all other programs doing regular read() may also be fooled about the file content, and this idea surprised me. On the other hand, after a write() other programs also see the new content without a guarantee that this content corresponds with what is on the disk...

No, read() will see the modified pagecache data immediately, apart from CPU cache coherency effects.

Is not this quite a big security hole?

cat evil_data > /tmp/sign.me [Okay, evil_data probably have to
contain lot of zeroes?]
sync, fill disk or wait for someone to fill disk completely

attempt to write good_data to /tmp/sign.me using mmap

"Hey, root, see what /tmp/sign.me contains, can you make it suid?"

root reads /tmp/sign.me, and sees it is good.

root does chown root.root /tmp/sign.me; chmod 4755 /tmp/sign.me

kernel realizes that there's not enough disk space, and discard changes, therefore /tmp/sign.me reverts to previous, evil, content.

Pavel

Signature · 1 line
When do you have heart between your knees?
#7 lore raw
Quoted 12 lines · Pavel Machek
Quoted 2 lines

No, read() will see the modified pagecache data immediately, apart from CPU cache coherency effects.

Is not this quite a big security hole?

cat evil_data > /tmp/sign.me [Okay, evil_data probably have to
contain lot of zeroes?]
sync, fill disk or wait for someone to fill disk completely

attempt to write good_data to /tmp/sign.me using mmap

Quoted 9 lines

"Hey, root, see what /tmp/sign.me contains, can you make it suid?"

root reads /tmp/sign.me, and sees it is good.

root does chown root.root /tmp/sign.me; chmod 4755 /tmp/sign.me

kernel realizes that there's not enough disk space, and discard changes, therefore /tmp/sign.me reverts to previous, evil, content.

root would have to make that change while user has the file open, and should welcome the subsequent unleashing of evil content as a valuable lesson.

#8 lore raw

Hi!

Quoted 19 lines
Quoted 19 lines
Quoted 3 lines

No, read() will see the modified pagecache data immediately, apart from CPU
cache coherency effects.

Is not this quite a big security hole?

cat evil_data > /tmp/sign.me [Okay, evil_data probably have to
contain lot of zeroes?]
sync, fill disk or wait for someone to fill disk completely

attempt to write good_data to /tmp/sign.me using mmap

"Hey, root, see what /tmp/sign.me contains, can you make it suid?"

root reads /tmp/sign.me, and sees it is good.

root does chown root.root /tmp/sign.me; chmod 4755 /tmp/sign.me

Quoted 7 lines
Quoted 3 lines

kernel realizes that there's not enough disk space, and discard changes, therefore /tmp/sign.me reverts to previous, evil, content.

root would have to make that change while user has the file open, and should welcome the subsequent unleashing of evil content as a valuable lesson.

Really? I thought that writeback is not synchronous at close() time.... Hmm.... It probably could be in case of mmap....

It is still pretty unexpected. Like "root sees you have that file open, so he stops you via ptrace".... but ok....
Pavel

#9 lore raw
Quoted 4 lines · Pavel Machek
Quoted 4 lines
Quoted 3 lines

kernel realizes that there's not enough disk space, and discard changes, therefore /tmp/sign.me reverts to previous, evil, content.

Quoted 8 lines
Quoted 3 lines

root would have to make that change while user has the file open, and should welcome the subsequent unleashing of evil content as a valuable lesson.

Really? I thought that writeback is not synchronous at close() time.... Hmm.... It probably could be in case of mmap....

writeback isn't, but the pages will get marked dirty at unmap. But I think I am wrong actually - I don't actually see why the user would have to have the file open.

Quoted 2 lines

It is still pretty unexpected. Like "root sees you have that file open, so he stops you via ptrace".... but ok....

Or maybe

cp /tmp/sign.me ~/
chown ... ~/sign.me
chmod ... ~/sign.me
mv ~/sign.me /tmp/signed

#10 lore raw
Quoted 3 lines · Nick Piggin

writeback isn't, but the pages will get marked dirty at unmap. But I think I am wrong actually - I don't actually see why the user would have to have the file open.

Dirty memory "limits" have no force as applied to mmap() IO, which is not a pretty state of affairs with respect to various attempts the VM makes at mitigating data structure proliferation associated with dirty data.

-- wli

#11 lore raw
Quoted 7 lines · William Lee Irwin III
Quoted 3 lines · Nick Piggin

writeback isn't, but the pages will get marked dirty at unmap. But I think I am wrong actually - I don't actually see why the user would have to have the file open.

Quoted 5 lines

Dirty memory "limits" have no force as applied to mmap() IO, which is not a pretty state of affairs with respect to various attempts the VM makes at mitigating data structure proliferation associated with dirty data.

Yeah I know. data structure proliferation and just the simple fact that it can't immediately be freed is a problem.

What is the alternative? Take a fault every time we write to a clean, mmapped page?

#12 lore raw
Quoted 4 lines · William Lee Irwin III
Quoted 4 lines

Dirty memory "limits" have no force as applied to mmap() IO, which is not a pretty state of affairs with respect to various attempts the VM makes at mitigating data structure proliferation associated with dirty data.

Quoted 4 lines · Nick Piggin

Yeah I know. data structure proliferation and just the simple fact that it can't immediately be freed is a problem.
What is the alternative? Take a fault every time we write to a clean, mmapped page?

That's the only option I'm now aware of. I suspect it may make sense to try it to get a notion of just how large the performance impact is so it's cost can be properly weighed against the expected stability benefit. But it's also worth noting that with some care the additional fault may be circumvented in some performance-relevant instances.

-- wli

#13 lore raw
In reply to Andrew Morton
Quoted 8 lines · Andrey Savochkin
Quoted 8 lines

Let's suppose an mmap'ed (SHARED, RW) file has a hole. AFAICS, we allow to dirty the file pages without allocating the space for the hole - filemap_nopage just "reads" the page filling it with zeroes, and nothing is done about the on-disk data until writepage. So, if the page can't be written to disk (no space), the dirty data just stays in the pagecache. The data can be read or seen via mmap, but it isn't and never be on disk. The pagecache stays unsynchronized with the on-disk content forever.

Quoted 8 lines · Andrew Morton

The kernel will make one attampt to write the data to disk. If that write hits ENOSPC, the page is not redirtied (ie: the data can be lost). When that write hits ENOSPC an error flag is set in the address_space and that will be returned from a subsequent msync(). The application will then need to do something about it.
If your application doesn't msync() the memory then it doesn't care about its data anyway. If your application _does_ msync the pages then we reliably report errors.

msync(p, sz, MS_ASYNC) only does set_page_dirty() at the moment and returns 0 unconditionally AFAICT, so things are stuck blocking and waiting for disk to reap the status of the IO at all. Maybe if that worked the fault handling wouldn't be as important. Maybe we should be reaping AS_EIO and/or AS_ENOSPC in the MS_ASYNC case, or wherever it is we stash the fact those IO errors ever happened. I'm also not sure what people think would be the right way to kick off IO in the background there, as trying to kmalloc() a workqueue element, then doing schedule_work() on it has resource management issues, but forcing userspace to block on the IO to ensure it's been initiated at all defeats the point of it.

Quoted 3 lines · Andrey Savochkin
Quoted 3 lines

Is it the intended behavior?
Shouldn't we call the filesystem to fill the hole at the moment of the first write access?

Quoted 7 lines · Andrew Morton

That would be a retrograde step - it would be nice to move in the other
direction: perform disk allocation at writeback time rather than at write() time, even for regular write() data. To do that we (probably) need space reservation APIs. And yes, we perhaps could reserve space in the filesystem when that page is first written to.
But then what would we do if there's no space? SIGBUS? SIGSEGV? Inappropriate. SIGENOSPC?

I believe SIGBUS is conventional, though you seem to be leaning toward solutions outside the fault path. I presume the "You're screwed without msync(2)" bit is standards-conformant.

-- wli

#14 lore raw
Quoted 11 lines · William Lee Irwin III

msync(p, sz, MS_ASYNC) only does set_page_dirty() at the moment and returns 0 unconditionally AFAICT, so things are stuck blocking and waiting for disk to reap the status of the IO at all. Maybe if that worked the fault handling wouldn't be as important. Maybe we should be reaping AS_EIO and/or AS_ENOSPC in the MS_ASYNC case, or wherever it is we stash the fact those IO errors ever happened. I'm also not sure what people think would be the right way to kick off IO in the background there, as trying to kmalloc() a workqueue element, then doing schedule_work() on it has resource management issues, but forcing userspace to block on the IO to ensure it's been initiated at all defeats the point of it.

And, interestingly, the only user of the result of set_page_dirty() is redirty_page_for_writepage(), whose results are ignored by all callers. It appears that something is amiss here, as failed reservations aren't reported until something attempts background writeback or IO syscalls. That is, it would seem that checking the results of set_page_dirty(), also called in the MS_ASYNC case, suffices, however, it does not return useful results in most (all?) cases, and nothing now checks its result.

The calling convention looks very very odd also; filemap_fdatawait() is the only apparent way to extract an ENOSPC result without calling the ->writepage() method directly, and this, instead of checking for things returning -ENOSPC as one would expect, does a rather odd thing, that is test_and_clear_bit(AS_ENOSPC, &mapping->flags), which will lose all but one of the results whenever there are multiple concurrent callers of it on a single inode. Worse yet, that can be legitimate, particularly when multiple tasks concurrently msync() disjoint subsets of a file's data.

-- wli

#15 lore raw
Quoted 14 lines · William Lee Irwin III
Quoted 11 lines · William Lee Irwin III

msync(p, sz, MS_ASYNC) only does set_page_dirty() at the moment and returns 0 unconditionally AFAICT, so things are stuck blocking and waiting for disk to reap the status of the IO at all. Maybe if that worked the fault handling wouldn't be as important. Maybe we should be reaping AS_EIO and/or AS_ENOSPC in the MS_ASYNC case, or wherever it is we stash the fact those IO errors ever happened. I'm also not sure what people think would be the right way to kick off IO in the background there, as trying to kmalloc() a workqueue element, then doing schedule_work() on it has resource management issues, but forcing userspace to block on the IO to ensure it's been initiated at all defeats the point of it.

Quoted 7 lines

And, interestingly, the only user of the result of set_page_dirty() is redirty_page_for_writepage(), whose results are ignored by all callers. It appears that something is amiss here, as failed reservations aren't reported until something attempts background writeback or IO syscalls. That is, it would seem that checking the results of set_page_dirty(), also called in the MS_ASYNC case, suffices, however, it does not return useful results in most (all?) cases, and nothing now checks its result.

Yes, the non-void return value from set_page_dirty() is a holdover from my very early allocate-on-flush patches, wherein set_page_dirty() did indeed reserve space in the filesystem.

Quoted 9 lines

The calling convention looks very very odd also; filemap_fdatawait() is the only apparent way to extract an ENOSPC result without calling the ->writepage() method directly, and this, instead of checking for things returning -ENOSPC as one would expect, does a rather odd thing, that is test_and_clear_bit(AS_ENOSPC, &mapping->flags), which will lose all but one of the results whenever there are multiple concurrent callers of it on a single inode. Worse yet, that can be legitimate, particularly when multiple tasks concurrently msync() disjoint subsets of a file's data.

Yes. But at least _someone_ gets told that there was an ENOSPC/EIO. What are the alternatives?

#16 lore raw
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines

And, interestingly, the only user of the result of set_page_dirty() is redirty_page_for_writepage(), whose results are ignored by all callers. It appears that something is amiss here, as failed reservations aren't reported until something attempts background writeback or IO syscalls. That is, it would seem that checking the results of set_page_dirty(), also called in the MS_ASYNC case, suffices, however, it does not return useful results in most (all?) cases, and nothing now checks its result.

Quoted 3 lines · Andrew Morton

Yes, the non-void return value from set_page_dirty() is a holdover from my very early allocate-on-flush patches, wherein set_page_dirty() did indeed reserve space in the filesystem.

Supposing one maintained upper and lower bounds on reserved space the best it appears to be able to do is a check on the lower bound and opportunistically add to the upper bound, as it's nonblocking. If the callers could be given hints to back out of their locks and retry reservations while blocking, that may do. filemap_fdatawait() can, but there are a lot of bizarre callers, e.g. fs/hfsplus/bnode.c, and no one maintains that kind of information.

Quoted 8 lines · William Lee Irwin III
Quoted 8 lines

The calling convention looks very very odd also; filemap_fdatawait() is the only apparent way to extract an ENOSPC result without calling the ->writepage() method directly, and this, instead of checking for things returning -ENOSPC as one would expect, does a rather odd thing, that is test_and_clear_bit(AS_ENOSPC, &mapping->flags), which will lose all but one of the results whenever there are multiple concurrent callers of it on a single inode. Worse yet, that can be legitimate, particularly when multiple tasks concurrently msync() disjoint subsets of a file's data.

Quoted 2 lines · Andrew Morton

Yes. But at least _someone_ gets told that there was an ENOSPC/EIO. What are the alternatives?

It seems more like a property of the sb, so referring things to ->i_sb and flagging the condition in there may make some sense. But a worse problem with all this is that the wrong one may catch the error. e.g. one process doing mmap() IO writes to a hole on a full fs, one process doing mmap() IO writes to already-allocated blocks, both block, AS_ENOSPC is set on behalf of the writer to the hole, and the writer to the already-allocated blocks reaps it. The only IO codepath that runs out of context from the submitting processes without alternative methods of returning the error is vmscan.c, so in principle returning errors to callers should work. But converting fs drivers to doing something for set_page_dirty() et al to report or even propagating -ENOSPC back from all of the ->writepage() and ->writepages() callsites looks painful. And supposing one moved the ENOSPC flag to the sb the events that would clear it aren't now trapped by anything.

Hmm. More questions than answers, again. Let me know if there are any fs/ or mm/ sweeps you want done for ENOSPC-relevant things (not necessarily any of the above).

-- wli