fs/ceph/addr: always call ceph_shift_unused_folios_left()

PATCH In review Max Kellermann 3 participants 12 messages Aug 27, 2025 – Sep 8, 2025

Subsystems (2)

Stable branch Ceph client

lore

Share & tools

lore

Max Kellermann

author

2025-08-27 18:17 UTC #0 lore raw

PATCH 1/1 fs/ceph/addr: always call ceph_shift_unused_folios_left()

The function ceph_process_folio_batch() sets folio_batch entries to NULL, which is an illegal state. Before folio_batch_release() crashes due to this API violation, the function ceph_shift_unused_folios_left() is supposed to remove those NULLs from the array.

However, since commit ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method"), this shifting doesn't happen anymore because the "for" loop got moved to ceph_process_folio_batch(), and now the i variable that remains in ceph_writepages_start() doesn't get incremented anymore, making the shifting effectively unreachable much of the time.

Later, commit 1551ec61dc55 ("ceph: introduce ceph_submit_write() method") added more preconditions for doing the shift, replacing the i check (with something that is still just as broken):

- if ceph_process_folio_batch() fails, shifting never happens

- if ceph_move_dirty_page_in_page_array() was never called (because ceph_process_folio_batch() has returned early for some of various reasons), shifting never happens

- if processed_in_fbatch is zero (because ceph_process_folio_batch() has returned early for some of the reasons mentioned above or because ceph_move_dirty_page_in_page_array() has failed), shifting never happens

Since those two commits, any problem in ceph_process_folio_batch() could crash the kernel, e.g. this way:

 BUG: kernel NULL pointer dereference, address: 0000000000000034
 #PF: supervisor write access in kernel mode
 #PF: error_code(0x0002) - not-present page
 PGD 0 P4D 0
 Oops: Oops: 0002 [#1] SMP NOPTI
 CPU: 172 UID: 0 PID: 2342707 Comm: kworker/u778:8 Not tainted 6.15.10-cm4all1-es #714 NONE
 Hardware name: Dell Inc. PowerEdge R7615/0G9DHV, BIOS 1.6.10 12/08/2023
 Workqueue: writeback wb_workfn (flush-ceph-1)
 RIP: 0010:folios_put_refs+0x85/0x140
 Code: 83 c5 01 39 e8 7e 76 48 63 c5 49 8b 5c c4 08 b8 01 00 00 00 4d 85 ed 74 05 41 8b 44 ad 00 48 8b 15 b0 >
 RSP: 0018:ffffb880af8db778 EFLAGS: 00010207
 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000003
 RDX: ffffe377cc3b0000 RSI: 0000000000000000 RDI: ffffb880af8db8c0
 RBP: 0000000000000000 R08: 000000000000007d R09: 000000000102b86f
 R10: 0000000000000001 R11: 00000000000000ac R12: ffffb880af8db8c0
 R13: 0000000000000000 R14: 0000000000000000 R15: ffff9bd262c97000
 FS:  0000000000000000(0000) GS:ffff9c8efc303000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000034 CR3: 0000000160958004 CR4: 0000000000770ef0
 PKRU: 55555554
 Call Trace:
  <TASK>
  ceph_writepages_start+0xeb9/0x1410

The crash can be reproduced easily by changing the ceph_check_page_before_write() return value to -E2BIG.

(Interestingly, the crash happens only if huge_zero_folio has already been allocated; without huge_zero_folio, is_huge_zero_folio(NULL) returns true and folios_put_refs() skips NULL entries instead of dereferencing them. That makes reproducing the bug somewhat unreliable. See
https://lore.kernel.org/[email protected]
for a discussion of this detail.)

My suggestion is to move the ceph_shift_unused_folios_left() to right after ceph_process_folio_batch() to ensure it always gets called to fix up the illegal folio_batch state.

Fixes ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method")

Link https://lore.kernel.org/ceph-devel/[email protected]/ open here

Cc [email protected]

Signed-off-by Max Kellermann <[email protected]>

---
 fs/ceph/addr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

fs/ceph/addr.c +1 -2

index 8b202d789e93..8bc66b45dade 100644
@@ -1687,6 +1687,7 @@ static int ceph_writepages_start(struct address_space *mapping,
 
 process_folio_batch:
 		rc = ceph_process_folio_batch(mapping, wbc, &ceph_wbc);
+		ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
 		if (rc)
 			goto release_folios;

@@ -1695,8 +1696,6 @@ static int ceph_writepages_start(struct address_space *mapping,
 			goto release_folios;
 
 		if (ceph_wbc.processed_in_fbatch) {
-			ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
-
 			if (folio_batch_count(&ceph_wbc.fbatch) == 0 &&
 			    ceph_wbc.locked_pages < ceph_wbc.max_pages) {
 				doutc(cl, "reached end fbatch, trying for more\n");

Signature · 1 line

2.47.2

Viacheslav Dubeyko

2025-08-27 19:07 UTC #1 lore raw

Quoted 91 lines · Max Kellermann

The function ceph_process_folio_batch() sets folio_batch entries to NULL, which is an illegal state. Before folio_batch_release() crashes due to this API violation, the function ceph_shift_unused_folios_left() is supposed to remove those NULLs from the array.
However, since commit ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method"), this shifting doesn't happen anymore because the "for" loop got moved to ceph_process_folio_batch(), and now the i variable that remains in ceph_writepages_start() doesn't get incremented anymore, making the shifting effectively unreachable much of the time.
Later, commit 1551ec61dc55 ("ceph: introduce ceph_submit_write() method") added more preconditions for doing the shift, replacing the i check (with something that is still just as broken):
- if ceph_process_folio_batch() fails, shifting never happens
- if ceph_move_dirty_page_in_page_array() was never called (because ceph_process_folio_batch() has returned early for some of various reasons), shifting never happens
- if processed_in_fbatch is zero (because ceph_process_folio_batch() has returned early for some of the reasons mentioned above or because ceph_move_dirty_page_in_page_array() has failed), shifting never happens
Since those two commits, any problem in ceph_process_folio_batch() could crash the kernel, e.g. this way:
 BUG: kernel NULL pointer dereference, address: 0000000000000034
 #PF: supervisor write access in kernel mode
 #PF: error_code(0x0002) - not-present page
 PGD 0 P4D 0
 Oops: Oops: 0002 [#1] SMP NOPTI
 CPU: 172 UID: 0 PID: 2342707 Comm: kworker/u778:8 Not tainted 6.15.10-cm4all1-es #714 NONE
 Hardware name: Dell Inc. PowerEdge R7615/0G9DHV, BIOS 1.6.10 12/08/2023
 Workqueue: writeback wb_workfn (flush-ceph-1)
 RIP: 0010:folios_put_refs+0x85/0x140
 Code: 83 c5 01 39 e8 7e 76 48 63 c5 49 8b 5c c4 08 b8 01 00 00 00 4d 85 ed 74 05 41 8b 44 ad 00 48 8b 15 b0 >
 RSP: 0018:ffffb880af8db778 EFLAGS: 00010207
 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000003
 RDX: ffffe377cc3b0000 RSI: 0000000000000000 RDI: ffffb880af8db8c0
 RBP: 0000000000000000 R08: 000000000000007d R09: 000000000102b86f
 R10: 0000000000000001 R11: 00000000000000ac R12: ffffb880af8db8c0
 R13: 0000000000000000 R14: 0000000000000000 R15: ffff9bd262c97000
 FS:  0000000000000000(0000) GS:ffff9c8efc303000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000034 CR3: 0000000160958004 CR4: 0000000000770ef0
 PKRU: 55555554
 Call Trace:
  <TASK>
  ceph_writepages_start+0xeb9/0x1410
The crash can be reproduced easily by changing the ceph_check_page_before_write() return value to -E2BIG.
(Interestingly, the crash happens only if huge_zero_folio has already been allocated; without huge_zero_folio, is_huge_zero_folio(NULL) returns true and folios_put_refs() skips NULL entries instead of dereferencing them. That makes reproducing the bug somewhat unreliable. See
https://lore.kernel.org/[email protected]
for a discussion of this detail.)
My suggestion is to move the ceph_shift_unused_folios_left() to right after ceph_process_folio_batch() to ensure it always gets called to fix up the illegal folio_batch state.
Fixes ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method")
Link https://lore.kernel.org/ceph-devel/[email protected]/ open here
Cc [email protected]
Signed-off-by Max Kellermann <[email protected]>
---
 fs/ceph/addr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/ceph/addr.c +1 -0
index 8b202d789e93..8bc66b45dade 100644
@@ -1687,6 +1687,7 @@ static int ceph_writepages_start(struct address_space *mapping,
 
 process_folio_batch:
 		rc = ceph_process_folio_batch(mapping, wbc, &ceph_wbc);
+		ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
 		if (rc)
 			goto release_folios;
 

fs/ceph/addr.c static int ceph_writepages_start(struct address_space *mapping, · line 1701

Quoted 8 lines

fs/ceph/addr.c +0 -2

@@ -1695,8 +1696,6 @@ static int ceph_writepages_start(struct address_space *mapping,
 			goto release_folios;
 
 		if (ceph_wbc.processed_in_fbatch) {
-			ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
-
 			if (folio_batch_count(&ceph_wbc.fbatch) == 0 &&
 			    ceph_wbc.locked_pages < ceph_wbc.max_pages) {
 				doutc(cl, "reached end fbatch, trying for more\n");

Let us try to reproduce the issue and to test the patch.

Thanks,
Slava.

Viacheslav Dubeyko

2025-08-28 18:55 UTC #2 lore raw

In reply to Max Kellermann

Quoted 55 lines · Max Kellermann

The function ceph_process_folio_batch() sets folio_batch entries to NULL, which is an illegal state. Before folio_batch_release() crashes due to this API violation, the function ceph_shift_unused_folios_left() is supposed to remove those NULLs from the array.
However, since commit ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method"), this shifting doesn't happen anymore because the "for" loop got moved to ceph_process_folio_batch(), and now the i variable that remains in ceph_writepages_start() doesn't get incremented anymore, making the shifting effectively unreachable much of the time.
Later, commit 1551ec61dc55 ("ceph: introduce ceph_submit_write() method") added more preconditions for doing the shift, replacing the i check (with something that is still just as broken):
- if ceph_process_folio_batch() fails, shifting never happens
- if ceph_move_dirty_page_in_page_array() was never called (because ceph_process_folio_batch() has returned early for some of various reasons), shifting never happens
- if processed_in_fbatch is zero (because ceph_process_folio_batch() has returned early for some of the reasons mentioned above or because ceph_move_dirty_page_in_page_array() has failed), shifting never happens
Since those two commits, any problem in ceph_process_folio_batch() could crash the kernel, e.g. this way:
 BUG: kernel NULL pointer dereference, address: 0000000000000034
 #PF: supervisor write access in kernel mode
 #PF: error_code(0x0002) - not-present page
 PGD 0 P4D 0
 Oops: Oops: 0002 [#1] SMP NOPTI
 CPU: 172 UID: 0 PID: 2342707 Comm: kworker/u778:8 Not tainted 6.15.10-cm4all1-es #714 NONE
 Hardware name: Dell Inc. PowerEdge R7615/0G9DHV, BIOS 1.6.10 12/08/2023
 Workqueue: writeback wb_workfn (flush-ceph-1)
 RIP: 0010:folios_put_refs+0x85/0x140
 Code: 83 c5 01 39 e8 7e 76 48 63 c5 49 8b 5c c4 08 b8 01 00 00 00 4d 85 ed 74 05 41 8b 44 ad 00 48 8b 15 b0 >
 RSP: 0018:ffffb880af8db778 EFLAGS: 00010207
 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000003
 RDX: ffffe377cc3b0000 RSI: 0000000000000000 RDI: ffffb880af8db8c0
 RBP: 0000000000000000 R08: 000000000000007d R09: 000000000102b86f
 R10: 0000000000000001 R11: 00000000000000ac R12: ffffb880af8db8c0
 R13: 0000000000000000 R14: 0000000000000000 R15: ffff9bd262c97000
 FS:  0000000000000000(0000) GS:ffff9c8efc303000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000034 CR3: 0000000160958004 CR4: 0000000000770ef0
 PKRU: 55555554
 Call Trace:
  <TASK>
  ceph_writepages_start+0xeb9/0x1410

Quoted 3 lines

The crash can be reproduced easily by changing the ceph_check_page_before_write() return value to -E2BIG.

I cannot reproduce the crash/issue. If ceph_check_page_before_write() returns -E2BIG, then nothing happens. There is no crush and no write operations could be processed by file system driver anymore. So, it doesn't look like recipe to reproduce the issue. I cannot confirm that the patch fixes the issue without clear way to reproduce the issue.

Could you please provide more clear explanation of the issue reproduction path?

Thanks,
Slava.

Quoted 41 lines

(Interestingly, the crash happens only if huge_zero_folio has already been allocated; without huge_zero_folio, is_huge_zero_folio(NULL) returns true and folios_put_refs() skips NULL entries instead of dereferencing them. That makes reproducing the bug somewhat unreliable. See
https://lore.kernel.org/[email protected]
for a discussion of this detail.)
My suggestion is to move the ceph_shift_unused_folios_left() to right after ceph_process_folio_batch() to ensure it always gets called to fix up the illegal folio_batch state.
Fixes ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method")
Link https://lore.kernel.org/ceph-devel/[email protected]/ open here
Cc [email protected]
Signed-off-by Max Kellermann <[email protected]>
---
 fs/ceph/addr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/ceph/addr.c +1 -2
index 8b202d789e93..8bc66b45dade 100644
@@ -1687,6 +1687,7 @@ static int ceph_writepages_start(struct address_space *mapping,
 
 process_folio_batch:
 		rc = ceph_process_folio_batch(mapping, wbc, &ceph_wbc);
+		ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
 		if (rc)
 			goto release_folios;
 
@@ -1695,8 +1696,6 @@ static int ceph_writepages_start(struct address_space *mapping,
 			goto release_folios;
 
 		if (ceph_wbc.processed_in_fbatch) {
-			ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
-
 			if (folio_batch_count(&ceph_wbc.fbatch) == 0 &&
 			    ceph_wbc.locked_pages < ceph_wbc.max_pages) {
 				doutc(cl, "reached end fbatch, trying for more\n");

Ilya Dryomov

maintainer

2025-08-28 19:05 UTC #3 lore raw

Quoted 61 lines · Viacheslav Dubeyko

Quoted 55 lines · Max Kellermann
The function ceph_process_folio_batch() sets folio_batch entries to NULL, which is an illegal state. Before folio_batch_release() crashes due to this API violation, the function ceph_shift_unused_folios_left() is supposed to remove those NULLs from the array.
However, since commit ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method"), this shifting doesn't happen anymore because the "for" loop got moved to ceph_process_folio_batch(), and now the i variable that remains in ceph_writepages_start() doesn't get incremented anymore, making the shifting effectively unreachable much of the time.
Later, commit 1551ec61dc55 ("ceph: introduce ceph_submit_write() method") added more preconditions for doing the shift, replacing the i check (with something that is still just as broken):
- if ceph_process_folio_batch() fails, shifting never happens
- if ceph_move_dirty_page_in_page_array() was never called (because ceph_process_folio_batch() has returned early for some of various reasons), shifting never happens
- if processed_in_fbatch is zero (because ceph_process_folio_batch() has returned early for some of the reasons mentioned above or because ceph_move_dirty_page_in_page_array() has failed), shifting never happens
Since those two commits, any problem in ceph_process_folio_batch() could crash the kernel, e.g. this way:
 BUG: kernel NULL pointer dereference, address: 0000000000000034
 #PF: supervisor write access in kernel mode
 #PF: error_code(0x0002) - not-present page
 PGD 0 P4D 0
 Oops: Oops: 0002 [#1] SMP NOPTI
 CPU: 172 UID: 0 PID: 2342707 Comm: kworker/u778:8 Not tainted 6.15.10-cm4all1-es #714 NONE
 Hardware name: Dell Inc. PowerEdge R7615/0G9DHV, BIOS 1.6.10 12/08/2023
 Workqueue: writeback wb_workfn (flush-ceph-1)
 RIP: 0010:folios_put_refs+0x85/0x140
 Code: 83 c5 01 39 e8 7e 76 48 63 c5 49 8b 5c c4 08 b8 01 00 00 00 4d 85 ed 74 05 41 8b 44 ad 00 48 8b 15 b0 >
 RSP: 0018:ffffb880af8db778 EFLAGS: 00010207
 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000003
 RDX: ffffe377cc3b0000 RSI: 0000000000000000 RDI: ffffb880af8db8c0
 RBP: 0000000000000000 R08: 000000000000007d R09: 000000000102b86f
 R10: 0000000000000001 R11: 00000000000000ac R12: ffffb880af8db8c0
 R13: 0000000000000000 R14: 0000000000000000 R15: ffff9bd262c97000
 FS:  0000000000000000(0000) GS:ffff9c8efc303000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000034 CR3: 0000000160958004 CR4: 0000000000770ef0
 PKRU: 55555554
 Call Trace:
  <TASK>
  ceph_writepages_start+0xeb9/0x1410
Quoted 3 lines
The crash can be reproduced easily by changing the ceph_check_page_before_write() return value to -E2BIG.

Quoted 7 lines

I cannot reproduce the crash/issue. If ceph_check_page_before_write() returns -E2BIG, then nothing happens. There is no crush and no write operations could be processed by file system driver anymore. So, it doesn't look like recipe to reproduce the issue. I cannot confirm that the patch fixes the issue without clear way to reproduce the issue.
Could you please provide more clear explanation of the issue reproduction path?

Hi Slava,

Was this bit taken into account?

Thanks,

Ilya

Viacheslav Dubeyko

2025-08-28 19:08 UTC #4 lore raw

Quoted 75 lines · Ilya Dryomov

Quoted 61 lines · Viacheslav Dubeyko
Quoted 55 lines · Max Kellermann
The function ceph_process_folio_batch() sets folio_batch entries to NULL, which is an illegal state. Before folio_batch_release() crashes due to this API violation, the function ceph_shift_unused_folios_left() is supposed to remove those NULLs from the array.
However, since commit ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method"), this shifting doesn't happen anymore because the "for" loop got moved to ceph_process_folio_batch(), and now the i variable that remains in ceph_writepages_start() doesn't get incremented anymore, making the shifting effectively unreachable much of the time.
Later, commit 1551ec61dc55 ("ceph: introduce ceph_submit_write() method") added more preconditions for doing the shift, replacing the i check (with something that is still just as broken):
- if ceph_process_folio_batch() fails, shifting never happens
- if ceph_move_dirty_page_in_page_array() was never called (because ceph_process_folio_batch() has returned early for some of various reasons), shifting never happens
- if processed_in_fbatch is zero (because ceph_process_folio_batch() has returned early for some of the reasons mentioned above or because ceph_move_dirty_page_in_page_array() has failed), shifting never happens
Since those two commits, any problem in ceph_process_folio_batch() could crash the kernel, e.g. this way:
 BUG: kernel NULL pointer dereference, address: 0000000000000034
 #PF: supervisor write access in kernel mode
 #PF: error_code(0x0002) - not-present page
 PGD 0 P4D 0
 Oops: Oops: 0002 [#1] SMP NOPTI
 CPU: 172 UID: 0 PID: 2342707 Comm: kworker/u778:8 Not tainted 6.15.10-cm4all1-es #714 NONE
 Hardware name: Dell Inc. PowerEdge R7615/0G9DHV, BIOS 1.6.10 12/08/2023
 Workqueue: writeback wb_workfn (flush-ceph-1)
 RIP: 0010:folios_put_refs+0x85/0x140
 Code: 83 c5 01 39 e8 7e 76 48 63 c5 49 8b 5c c4 08 b8 01 00 00 00 4d 85 ed 74 05 41 8b 44 ad 00 48 8b 15 b0 >
 RSP: 0018:ffffb880af8db778 EFLAGS: 00010207
 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000003
 RDX: ffffe377cc3b0000 RSI: 0000000000000000 RDI: ffffb880af8db8c0
 RBP: 0000000000000000 R08: 000000000000007d R09: 000000000102b86f
 R10: 0000000000000001 R11: 00000000000000ac R12: ffffb880af8db8c0
 R13: 0000000000000000 R14: 0000000000000000 R15: ffff9bd262c97000
 FS:  0000000000000000(0000) GS:ffff9c8efc303000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000034 CR3: 0000000160958004 CR4: 0000000000770ef0
 PKRU: 55555554
 Call Trace:
  <TASK>
  ceph_writepages_start+0xeb9/0x1410
Quoted 3 lines
The crash can be reproduced easily by changing the ceph_check_page_before_write() return value to -E2BIG.
Quoted 7 lines
I cannot reproduce the crash/issue. If ceph_check_page_before_write() returns -E2BIG, then nothing happens. There is no crush and no write operations could be processed by file system driver anymore. So, it doesn't look like recipe to reproduce the issue. I cannot confirm that the patch fixes the issue without clear way to reproduce the issue.
Could you please provide more clear explanation of the issue reproduction path?
Hi Slava,
Was this bit taken into account?

Quoted 9 lines

(Interestingly, the crash happens only if huge_zero_folio has already been allocated; without huge_zero_folio, is_huge_zero_folio(NULL) returns true and folios_put_refs() skips NULL entries instead of dereferencing them. That makes reproducing the bug somewhat unreliable. See
https://lore.kernel.org/[email protected]
for a discussion of this detail.)

Hi Ilya,

And which practical step of actions do you see to repeat and reproduce it? :)

Thanks,
Slava.

Max Kellermann

author

2025-08-28 21:37 UTC #5 lore raw

Quoted 1 line · Viacheslav Dubeyko

And which practical step of actions do you see to repeat and reproduce it? :)

Apply the patch in the link. Did you read that thread/patch?

Viacheslav Dubeyko

Reviewed-by Tested-by

2025-09-04 21:43 UTC #6 lore raw

Quoted 5 lines · Max Kellermann

Quoted 1 line · Viacheslav Dubeyko
And which practical step of actions do you see to repeat and reproduce it? :)
Apply the patch in the link. Did you read that thread/patch?

By applying the patch [1], enabling CONFIG_DEBUG_VM, and returning -E2BIG from ceph_check_page_before_write(), I was able to reproduce this warning:

 [  123.147833] ------------[ cut here ]------------
 [  123.147861] WARNING: CPU: 5 PID: 72 at ./include/linux/huge_mm.h:482
folios_put_refs+0x4c2/0x600
 [  123.147900] Modules linked in: intel_rapl_msr intel_rapl_common

intel_uncore_frequency_common intel_pmc_core pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec kvm_intel kvm irqbypass joydev polyval_clmulni ghash_clmulni_intel aesni_intel rapl input_leds psmouse i2c_piix4 vga16fb pata_acpi bochs vgastate i2c_smbus serio_raw floppy qemu_fw_cfg mac_hid sch_fq_codel rbd msr parport_pc ppdev lp parport efi_pstore

 [  123.147988] CPU: 5 UID: 0 PID: 72 Comm: kworker/u32:2 Not tainted 6.17.0-
rc4+ #9 PREEMPT(voluntary) 
 [  123.147995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.17.0-5.fc42 04/01/2014
 [  123.148002] Workqueue: writeback wb_workfn (flush-ceph-1)
 [  123.148021] RIP: 0010:folios_put_refs+0x4c2/0x600
 [  123.148031] Code: cc c6 db 05 0f 85 19 01 00 00 48 81 c4 b8 00 00 00 5b 41
5c 41 5d 41 5e 41 5f 5d 31 c0 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc <0f> 0b e9
1e fe ff ff e8 c2 fe 24 00 e9 da fb ff ff 4c 89 ef e8 b5
 [  123.148035] RSP: 0018:ffff888101c6f228 EFLAGS: 00010246
 [  123.148051] RAX: ffffed102038dea4 RBX: 0000000000000000 RCX:
0000000000000000
 [  123.148057] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
ffff888101c6f520
 [  123.148060] RBP: ffff888101c6f308 R08: 0000000000000000 R09:
0000000000000000
 [  123.148063] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
 [  123.148066] R13: ffff888101c6f520 R14: 0000000000000000 R15:
dffffc0000000000
 [  123.148069] FS:  0000000000000000(0000) GS:ffff88824a034000(0000)
knlGS:0000000000000000
 [  123.148072] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 [  123.148075] CR2: 0000700798000020 CR3: 0000000111b6a005 CR4:
0000000000772ef0
 [  123.148082] PKRU: 55555554
 [  123.148085] Call Trace:
 [  123.148088]  <TASK>
 [  123.148093]  ? __pfx_folios_put_refs+0x10/0x10
 [  123.148099]  ? __pfx_filemap_get_folios_tag+0x10/0x10
 [  123.148110]  __folio_batch_release+0x52/0xe0
 [  123.148115]  ceph_writepages_start+0x277a/0x45f0
 [  123.148129]  ? update_load_avg+0x1bd/0x1fe0
 [  123.148145]  ? dequeue_entity+0x3e5/0x1450
 [  123.148151]  ? ata_sff_qc_issue+0x443/0xa90
 [  123.148175]  ? kvm_sched_clock_read+0x11/0x20
 [  123.148198]  ? sched_clock_noinstr+0x9/0x10
 [  123.148203]  ? sched_clock+0x10/0x30
 [  123.148216]  ? __pfx_ceph_writepages_start+0x10/0x10
 [  123.148221]  ? psi_group_change+0x3fa/0x8a0
 [  123.148233]  ? __pfx_sched_clock_cpu+0x10/0x10
 [  123.148238]  ? set_next_entity+0x325/0xb40
 [  123.148245]  ? ncsi_channel_monitor.cold+0x36d/0x553
 [  123.148269]  ? __kasan_check_write+0x14/0x30
 [  123.148283]  ? _raw_spin_lock+0x82/0xf0
 [  123.148293]  ? __pfx__raw_spin_lock+0x10/0x10
 [  123.148298]  do_writepages+0x1e1/0x540
 [  123.148303]  ? do_writepages+0x1e1/0x540
 [  123.148308]  __writeback_single_inode+0xa7/0x940
 [  123.148312]  ? _raw_spin_unlock+0xe/0x40
 [  123.148315]  ? wbc_attach_and_unlock_inode+0x440/0x610
 [  123.148325]  ? __pfx_call_function_single_prep_ipi+0x10/0x10
 [  123.148336]  writeback_sb_inodes+0x563/0xe40
 [  123.148341]  ? __pfx_writeback_sb_inodes+0x10/0x10
 [  123.148348]  ? __pfx_move_expired_inodes+0x10/0x10
 [  123.148360]  __writeback_inodes_wb+0xbe/0x210
 [  123.148364]  wb_writeback+0x4e4/0x6f0
 [  123.148368]  ? __pfx_wb_writeback+0x10/0x10
 [  123.148416]  ? get_nr_dirty_inodes+0xdc/0x1e0
 [  123.148426]  wb_workfn+0x5a9/0xb30
 [  123.148430]  ? __pfx_wb_workfn+0x10/0x10
 [  123.148433]  ? __pfx___schedule+0x10/0x10
 [  123.148438]  ? __pfx__raw_spin_lock_irq+0x10/0x10
 [  123.148442]  process_one_work+0x611/0xe20
 [  123.148448]  ? __kasan_check_write+0x14/0x30
 [  123.148452]  worker_thread+0x7e3/0x1580
 [  123.148456]  ? __pfx_worker_thread+0x10/0x10
 [  123.148458]  kthread+0x381/0x7a0
 [  123.148463]  ? __pfx__raw_spin_lock_irq+0x10/0x10
 [  123.148466]  ? __pfx_kthread+0x10/0x10
 [  123.148468]  ? __kasan_check_write+0x14/0x30
 [  123.148471]  ? recalc_sigpending+0x160/0x220
 [  123.148478]  ? _raw_spin_unlock_irq+0xe/0x50
 [  123.148481]  ? calculate_sigpending+0x78/0xb0
 [  123.148484]  ? __pfx_kthread+0x10/0x10
 [  123.148487]  ret_from_fork+0x285/0x350
 [  123.148490]  ? __pfx_kthread+0x10/0x10
 [  123.148493]  ret_from_fork_asm+0x1a/0x30
 [  123.148499]  </TASK>
 [  123.148501] ---[ end trace 0000000000000000 ]---

The warning has been eliminated by applying suggested fix. The suggested patch has been tested by xfstests and no regression or issue has been detected.

Reviewed-by Viacheslav Dubeyko <[email protected]>

Tested-by Viacheslav Dubeyko <[email protected]>

Thanks,
Slava.

[1]
https://lore.kernel.org/all/[email protected]/

Max Kellermann

author

2025-09-05 03:41 UTC #7 lore raw

Quoted 2 lines · Viacheslav Dubeyko

By applying the patch [1], enabling CONFIG_DEBUG_VM, and returning -E2BIG from ceph_check_page_before_write(), I was able to reproduce this warning:

Thanks, I'm glad you could verify the bug and my fix. In case this wasn't clear: you saw just a warning, but this is usually a kernel crash due to NULL pointer dereference. If you only got a warning but no crash, it means your test VM does not use transparent huge pages (no huge_zero_folio allocated yet). In a real workload, the kernel would have crashed.

Viacheslav Dubeyko

2025-09-05 17:12 UTC #8 lore raw

Quoted 11 lines · Max Kellermann

Quoted 2 lines · Viacheslav Dubeyko
By applying the patch [1], enabling CONFIG_DEBUG_VM, and returning -E2BIG from ceph_check_page_before_write(), I was able to reproduce this warning:
Thanks, I'm glad you could verify the bug and my fix. In case this wasn't clear: you saw just a warning, but this is usually a kernel crash due to NULL pointer dereference. If you only got a warning but no crash, it means your test VM does not use transparent huge pages (no huge_zero_folio allocated yet). In a real workload, the kernel would have crashed.

I would like to reproduce the crash. But you've share only these steps. And it looks like that it's not the complete recipe. So, something was missing. If you could share more precise explanation of steps, it will be great.

Thanks,
Slava.

Max Kellermann

author

2025-09-05 18:35 UTC #9 lore raw

Quoted 9 lines · Viacheslav Dubeyko

Quoted 6 lines · Max Kellermann
Thanks, I'm glad you could verify the bug and my fix. In case this wasn't clear: you saw just a warning, but this is usually a kernel crash due to NULL pointer dereference. If you only got a warning but no crash, it means your test VM does not use transparent huge pages (no huge_zero_folio allocated yet). In a real workload, the kernel would have crashed.

Quoted 3 lines

I would like to reproduce the crash. But you've share only these steps. And it looks like that it's not the complete recipe. So, something was missing. If you could share more precise explanation of steps, it will be great.

The email you just cited explains the circumstances that are necessary for the crash to occur.

Let me repeat it for you: you have to ensure that huge_zero_folio gets allocated (or else the code that dereferences the NULL pointer and crashes gets skipped).

Got it now?

Viacheslav Dubeyko

2025-09-08 18:40 UTC #10 lore raw

Quoted 14 lines · Max Kellermann

Quoted 9 lines · Viacheslav Dubeyko
Quoted 6 lines · Max Kellermann
Thanks, I'm glad you could verify the bug and my fix. In case this wasn't clear: you saw just a warning, but this is usually a kernel crash due to NULL pointer dereference. If you only got a warning but no crash, it means your test VM does not use transparent huge pages (no huge_zero_folio allocated yet). In a real workload, the kernel would have crashed.
Quoted 3 lines
I would like to reproduce the crash. But you've share only these steps. And it looks like that it's not the complete recipe. So, something was missing. If you could share more precise explanation of steps, it will be great.

Quoted 8 lines

The email you just cited explains the circumstances that are necessary for the crash to occur.
Let me repeat it for you: you have to ensure that huge_zero_folio gets allocated (or else the code that dereferences the NULL pointer and crashes gets skipped).
Got it now?

Not yet. I would like to see the clear explanation of steps. Otherwise, the probability to reproduce the crash is equal to zero.

Thanks,
Slava.

Ilya Dryomov

maintainer

2025-09-08 10:32 UTC #11 lore raw

In reply to Max Kellermann

Quoted 101 lines · Max Kellermann

The function ceph_process_folio_batch() sets folio_batch entries to NULL, which is an illegal state. Before folio_batch_release() crashes due to this API violation, the function ceph_shift_unused_folios_left() is supposed to remove those NULLs from the array.
However, since commit ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method"), this shifting doesn't happen anymore because the "for" loop got moved to ceph_process_folio_batch(), and now the i variable that remains in ceph_writepages_start() doesn't get incremented anymore, making the shifting effectively unreachable much of the time.
Later, commit 1551ec61dc55 ("ceph: introduce ceph_submit_write() method") added more preconditions for doing the shift, replacing the i check (with something that is still just as broken):
- if ceph_process_folio_batch() fails, shifting never happens
- if ceph_move_dirty_page_in_page_array() was never called (because ceph_process_folio_batch() has returned early for some of various reasons), shifting never happens
- if processed_in_fbatch is zero (because ceph_process_folio_batch() has returned early for some of the reasons mentioned above or because ceph_move_dirty_page_in_page_array() has failed), shifting never happens
Since those two commits, any problem in ceph_process_folio_batch() could crash the kernel, e.g. this way:
 BUG: kernel NULL pointer dereference, address: 0000000000000034
 #PF: supervisor write access in kernel mode
 #PF: error_code(0x0002) - not-present page
 PGD 0 P4D 0
 Oops: Oops: 0002 [#1] SMP NOPTI
 CPU: 172 UID: 0 PID: 2342707 Comm: kworker/u778:8 Not tainted 6.15.10-cm4all1-es #714 NONE
 Hardware name: Dell Inc. PowerEdge R7615/0G9DHV, BIOS 1.6.10 12/08/2023
 Workqueue: writeback wb_workfn (flush-ceph-1)
 RIP: 0010:folios_put_refs+0x85/0x140
 Code: 83 c5 01 39 e8 7e 76 48 63 c5 49 8b 5c c4 08 b8 01 00 00 00 4d 85 ed 74 05 41 8b 44 ad 00 48 8b 15 b0 >
 RSP: 0018:ffffb880af8db778 EFLAGS: 00010207
 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000003
 RDX: ffffe377cc3b0000 RSI: 0000000000000000 RDI: ffffb880af8db8c0
 RBP: 0000000000000000 R08: 000000000000007d R09: 000000000102b86f
 R10: 0000000000000001 R11: 00000000000000ac R12: ffffb880af8db8c0
 R13: 0000000000000000 R14: 0000000000000000 R15: ffff9bd262c97000
 FS:  0000000000000000(0000) GS:ffff9c8efc303000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000034 CR3: 0000000160958004 CR4: 0000000000770ef0
 PKRU: 55555554
 Call Trace:
  <TASK>
  ceph_writepages_start+0xeb9/0x1410
The crash can be reproduced easily by changing the ceph_check_page_before_write() return value to -E2BIG.
(Interestingly, the crash happens only if huge_zero_folio has already been allocated; without huge_zero_folio, is_huge_zero_folio(NULL) returns true and folios_put_refs() skips NULL entries instead of dereferencing them. That makes reproducing the bug somewhat unreliable. See
https://lore.kernel.org/[email protected]
for a discussion of this detail.)
My suggestion is to move the ceph_shift_unused_folios_left() to right after ceph_process_folio_batch() to ensure it always gets called to fix up the illegal folio_batch state.
Fixes ce80b76dd327 ("ceph: introduce ceph_process_folio_batch() method")
Link https://lore.kernel.org/ceph-devel/[email protected]/ open here
Cc [email protected]
Signed-off-by Max Kellermann <[email protected]>
---
 fs/ceph/addr.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/ceph/addr.c +1 -3
index 8b202d789e93..8bc66b45dade 100644
@@ -1687,6 +1687,7 @@ static int ceph_writepages_start(struct address_space *mapping,
 
 process_folio_batch:
                rc = ceph_process_folio_batch(mapping, wbc, &ceph_wbc);
+               ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
                if (rc)
                        goto release_folios;
 
@@ -1695,8 +1696,6 @@ static int ceph_writepages_start(struct address_space *mapping,
                        goto release_folios;
 
                if (ceph_wbc.processed_in_fbatch) {
-                       ceph_shift_unused_folios_left(&ceph_wbc.fbatch);
-
                        if (folio_batch_count(&ceph_wbc.fbatch) == 0 &&
                            ceph_wbc.locked_pages < ceph_wbc.max_pages) {
                                doutc(cl, "reached end fbatch, trying for more\n");
--

Quoted 2 lines

2.47.2

Queued up for 6.17-rc6.

Thanks,

Ilya