Skip to contentSkip to conversation

xfs: add guard before freeing buffer log item

PATCH In review Marcelo Mendes Spessoto Junior 2 participants 4 messages Jul 31 – Aug 5 XFS filesystem

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs: add guard before freeing buffer log item

A buffer's write completion can race with the CIL walking the same checkpoint's item list during a forced shutdown's simulated commit callbacks. Whichever side reaches the item last should be the one to free it; right now completion frees it unconditionally, so the CIL walk can end up touching memory that's already gone.

Fixes d2fe5c4c8d25 ("xfs: rearrange code in xfs_buf_item.c")
Reported-by [email protected]
---
 fs/xfs/xfs_buf_item.c | 3 +++
 1 file changed, 3 insertions(+)
fs/xfs/xfs_buf_item.c +3 -0
index f4c5be67826e..c86c4387f52b 100644
@@ -1072,6 +1072,9 @@ void
 xfs_buf_item_done(
 	struct xfs_buf		*bp)
 {
+	if (atomic_read(&bp->b_log_item->bli_refcount) != 0)
+		return;
+
 	/*
 	 * If we are forcibly shutting down, this may well be off the AIL
 	 * already. That's because we simulate the log-committed callbacks to
Signature · 1 line
2.55.0
#1 lore raw

I've missed the original discussion about the issue
(https://lore.kernel.org/all/[email protected]/T/)
before submitting the fix. I will work on a v2 addressing the root causes, and this patch can be ignored.

Quoted 33 lines · Marcelo Mendes Spessoto Junior

A buffer's write completion can race with the CIL walking the same checkpoint's item list during a forced shutdown's simulated commit callbacks. Whichever side reaches the item last should be the one to free it; right now completion frees it unconditionally, so the CIL walk can end up touching memory that's already gone.

Fixes d2fe5c4c8d25 ("xfs: rearrange code in xfs_buf_item.c")
Reported-by [email protected]
---
 fs/xfs/xfs_buf_item.c | 3 +++
 1 file changed, 3 insertions(+)
fs/xfs/xfs_buf_item.c +3 -1
index f4c5be67826e..c86c4387f52b 100644
@@ -1072,6 +1072,9 @@ void
 xfs_buf_item_done(
        struct xfs_buf          *bp)
 {
+       if (atomic_read(&bp->b_log_item->bli_refcount) != 0)
+               return;
+
        /*
         * If we are forcibly shutting down, this may well be off the AIL
         * already. That's because we simulate the log-committed callbacks to
--

2.55.0

#2 lore raw
In reply to Marcelo Mendes Spessoto Junior
Quoted 20 lines · Marcelo Mendes Spessoto Junior

A buffer's write completion can race with the CIL walking the same checkpoint's item list during a forced shutdown's simulated commit callbacks. Whichever side reaches the item last should be the one to free it; right now completion frees it unconditionally, so the CIL walk can end up touching memory that's already gone.

Fixes d2fe5c4c8d25 ("xfs: rearrange code in xfs_buf_item.c")
Reported-by [email protected]
---
 fs/xfs/xfs_buf_item.c | 3 +++
 1 file changed, 3 insertions(+)
fs/xfs/xfs_buf_item.c +0 -0
index f4c5be67826e..c86c4387f52b 100644

fs/xfs/xfs_buf_item.c void · line 1076

Quoted 5 lines
fs/xfs/xfs_buf_item.c +2 -0
@@ -1072,6 +1072,9 @@ void
 xfs_buf_item_done(
 	struct xfs_buf		*bp)
 {
+	if (atomic_read(&bp->b_log_item->bli_refcount) != 0)
+		return;

No.

My conclusion to the original syzbot report was:

| As such, I think that the issue we need to understand here is why a
| write IO completion is being run from read IO context. That's the
| root cause of the UAF, not the commit the bisect landed on....

And if you look at what I posted yesterday about a different hack to this same code:

https://lore.kernel.org/linux-xfs/amvI_rVUlifIXoZA@dread/

which "fixes" a UAF in the CIL code reported by KASAN under syzbot conditions. Whilst the syzbot report quoted was different:

https://syzkaller.appspot.com/bug?extid=598a791b31c498b63c6b

The syzbot report indicates exactly the same UAF situation - a write IO completion being run from a read IO context causing the BLI to be freed incorrectly and then the CIL access to the BLI triggering a UAF. My conclusion to that one is:

| Given that syzbot has only reported this 5 times in 12 hours only on
| a 7.1-rc3 kernel, never before and never since, an external memory
| corruption bug that has since been fixed seems like the most like
| cause here.

All the recent reports of this issue (the cluster in mid May) are also only from v7.1-rc3 kernels....

So, yeah, people really need to stop trying to hack "fixes" into xfs_buf_item_done(). The UAF indicates that there has been an -incorrect free of the item- during IO completion. That's the issue that needs to be understood and fixed. Fundamentally breaking the refcounting model of the BLI to silence the downstream UAF that results from the incorrect free is not an acceptible solution.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 58 lines · Dave Chinner
Quoted 20 lines · Marcelo Mendes Spessoto Junior

A buffer's write completion can race with the CIL walking the same checkpoint's item list during a forced shutdown's simulated commit callbacks. Whichever side reaches the item last should be the one to free it; right now completion frees it unconditionally, so the CIL walk can end up touching memory that's already gone.

Fixes d2fe5c4c8d25 ("xfs: rearrange code in xfs_buf_item.c")
Reported-by [email protected]
---
 fs/xfs/xfs_buf_item.c | 3 +++
 1 file changed, 3 insertions(+)
fs/xfs/xfs_buf_item.c +0 -0
index f4c5be67826e..c86c4387f52b 100644

fs/xfs/xfs_buf_item.c void · line 1076

Quoted 5 lines
fs/xfs/xfs_buf_item.c +2 -0
@@ -1072,6 +1072,9 @@ void
 xfs_buf_item_done(
 	struct xfs_buf		*bp)
 {
+	if (atomic_read(&bp->b_log_item->bli_refcount) != 0)
+		return;

No.

My conclusion to the original syzbot report was:

| As such, I think that the issue we need to understand here is why a
| write IO completion is being run from read IO context. That's the
| root cause of the UAF, not the commit the bisect landed on....

And if you look at what I posted yesterday about a different hack to this same code:

https://lore.kernel.org/linux-xfs/amvI_rVUlifIXoZA@dread/

which "fixes" a UAF in the CIL code reported by KASAN under syzbot conditions. Whilst the syzbot report quoted was different:

https://syzkaller.appspot.com/bug?extid=598a791b31c498b63c6b

The syzbot report indicates exactly the same UAF situation - a write IO completion being run from a read IO context causing the BLI to be freed incorrectly and then the CIL access to the BLI triggering a UAF. My conclusion to that one is:

| Given that syzbot has only reported this 5 times in 12 hours only on
| a 7.1-rc3 kernel, never before and never since, an external memory
| corruption bug that has since been fixed seems like the most like
| cause here.

All the recent reports of this issue (the cluster in mid May) are also only from v7.1-rc3 kernels....

Quoted 6 lines

So, yeah, people really need to stop trying to hack "fixes" into xfs_buf_item_done(). The UAF indicates that there has been an -incorrect free of the item- during IO completion. That's the issue that needs to be understood and fixed. Fundamentally breaking the refcounting model of the BLI to silence the downstream UAF that results from the incorrect free is not an acceptible solution.

I could of followup thoughts. If the problem is that XBF_READ is not set when we expect it, we should probably try to catch a buffer that has neither XBF_READ or XBF_WRITE set at completion time. That would be a clear sign of external memory corruption, and so we could log an error, dump the state of the xfs_buf around the b_flags field to discover if it's just a b_flags issue or a larger corruption. We can then error out the IO without assuming that it is a WRITE IO, and let the caller handle it appropriately.

In this situation, if neither READ nor WRITE is set, we'll get the above debug. If we a completion where WRITE is set instead of READ, we'll get this same UAF report again.

We probably should check for any flag fields that should not be seen in IO completion are set (either unknown bits or bits that should never be seen at completion like XBF_TRYLOCK), that way we'd also capture corruptions that don't change READ/WRITE completions that trigger the UAF.

That will tell us more about what is going wrong here and hopefully help us isolate a potential cause...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]