memcg: Only record foreign writebacks with dirty pages when memcg is not disabled
Subsystems (2)
PATCH 1/1 memcg: Only record foreign writebacks with dirty pages when memcg is not disabled
In kdump kernel, memcg usually is disabled with 'cgroup_disable=memory' for saving memory. Now kdump kernel will always panic when dump vmcore to local disk:
BUG: kernel NULL pointer dereference, address: 0000000000000ab8 PGD 5fcab067 P4D 5fcab067 PUD 5ff73067 PMD 0 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 598 Comm: makedumpfile Not tainted 5.3.0+ #26 Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 RIP: 0010:mem_cgroup_track_foreign_dirty_slowpath+0x38/0x140 Code: 55 48 8b 2d 6a bd 12 01 53 4c 8b 67 38 0f 1f 44 00 00 49 8b 06 48 89 e9 31 db be ff ff ff ff 48 8b 38 49 8d 84 24 d0 0a 00 00 <48> 39 78 e8 74 5a 48 8b 50 f8 48 39 ca 79 0e 44 8b 00 41 83 f8 01 RSP: 0018:ffffbc9300817bd0 EFLAGS: 00010046 RAX: 0000000000000ad0 RBX: 0000000000000000 RCX: 00000000fffba227 RDX: fffffffffffffff8 RSI: 00000000ffffffff RDI: 0000000000000004 RBP: 00000000fffba227 R08: 0000000000030340 R09: ffffbc9300817d10 R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000000 R13: ffff999c1fa2bcf0 R14: ffff999c1f7bcc78 R15: ffffe51fc177a8c0 FS: 00007f84f14c2b80(0000) GS:ffff999c22200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000ab8 CR3: 000000005f564000 CR4: 00000000003406b0 Call Trace: __set_page_dirty+0x52/0xc0 iomap_set_page_dirty+0x50/0x90 iomap_write_end+0x6e/0x270 iomap_write_actor+0xce/0x170 ? iomap_write_end+0x270/0x270 iomap_apply+0xba/0x11e ? iomap_write_end+0x270/0x270 iomap_file_buffered_write+0x62/0x90 ? iomap_write_end+0x270/0x270 xfs_file_buffered_aio_write+0xca/0x320 [xfs] new_sync_write+0x12d/0x1d0 vfs_write+0xa5/0x1a0 ksys_write+0x59/0xd0 do_syscall_64+0x59/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f84f1093ab5
And this will corrupt the 1st kernel too with 'cgroup_disable=memory'.
From the trace and with debugging, it is pointing to commit 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing") which introduced this regression. Disabling memcg causes the null pointer dereference at uninitialized data in function mem_cgroup_track_foreign_dirty_slowpath().
Fix it by returning directly if memcg is disabled, but not trying to record the foreign writebacks with dirty pages.
Fixed: 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing")
--- include/linux/memcontrol.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/memcontrol.h +2 -1
@@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,static inline void mem_cgroup_track_foreign_dirty (struct page *page,struct bdi_writeback *wb) { -if (unlikely (&page->mem_cgroup->css != wb->memcg_css)) +if (unlikely (&page->mem_cgroup->css != wb->memcg_css) + && !mem_cgroup_disabled ())mem_cgroup_track_foreign_dirty_slowpath (page, wb); }
Signature · 1 line
2.17.2
Quoted 9 lines · Baoquan He
--- include/linux/memcontrol.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)include/linux/memcontrol.h +0
-0
include/linux/memcontrol.h void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, · line 1265
Quoted 6 lines
include/linux/memcontrol.h +2
-1@@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,static inline void mem_cgroup_track_foreign_dirty (struct page *page,struct bdi_writeback *wb) { -if (unlikely (&page->mem_cgroup->css != wb->memcg_css)) +if (unlikely (&page->mem_cgroup->css != wb->memcg_css) + && !mem_cgroup_disabled ())
Sorry, this is the draft patch I was testing. Later I think this had better be moved into mem_cgroup_track_foreign_dirty_slowpath().
Not very sure about this. Will send a v2 to match the patch log.
Quoted 6 lines
include/linux/memcontrol.h +0
-0mem_cgroup_track_foreign_dirty_slowpath (page, wb); }Signature · 1 line
2.17.2
PATCH v2 1/1 memcg: Only record foreign writebacks with dirty pages when memcg is not disabled
In reply to Baoquan HeIn kdump kernel, memcg usually is disabled with 'cgroup_disable=memory' for saving memory. Now kdump kernel will always panic when dump vmcore to local disk:
BUG: kernel NULL pointer dereference, address: 0000000000000ab8 PGD 5fcab067 P4D 5fcab067 PUD 5ff73067 PMD 0 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 598 Comm: makedumpfile Not tainted 5.3.0+ #26 Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 RIP: 0010:mem_cgroup_track_foreign_dirty_slowpath+0x38/0x140 Code: 55 48 8b 2d 6a bd 12 01 53 4c 8b 67 38 0f 1f 44 00 00 49 8b 06 48 89 e9 31 db be ff ff ff ff 48 8b 38 49 8d 84 24 d0 0a 00 00 <48> 39 78 e8 74 5a 48 8b 50 f8 48 39 ca 79 0e 44 8b 00 41 83 f8 01 RSP: 0018:ffffbc9300817bd0 EFLAGS: 00010046 RAX: 0000000000000ad0 RBX: 0000000000000000 RCX: 00000000fffba227 RDX: fffffffffffffff8 RSI: 00000000ffffffff RDI: 0000000000000004 RBP: 00000000fffba227 R08: 0000000000030340 R09: ffffbc9300817d10 R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000000 R13: ffff999c1fa2bcf0 R14: ffff999c1f7bcc78 R15: ffffe51fc177a8c0 FS: 00007f84f14c2b80(0000) GS:ffff999c22200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000ab8 CR3: 000000005f564000 CR4: 00000000003406b0 Call Trace: __set_page_dirty+0x52/0xc0 iomap_set_page_dirty+0x50/0x90 iomap_write_end+0x6e/0x270 iomap_write_actor+0xce/0x170 ? iomap_write_end+0x270/0x270 iomap_apply+0xba/0x11e ? iomap_write_end+0x270/0x270 iomap_file_buffered_write+0x62/0x90 ? iomap_write_end+0x270/0x270 xfs_file_buffered_aio_write+0xca/0x320 [xfs] new_sync_write+0x12d/0x1d0 vfs_write+0xa5/0x1a0 ksys_write+0x59/0xd0 do_syscall_64+0x59/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f84f1093ab5
And this will corrupt the 1st kernel too with 'cgroup_disable=memory'.
From the trace and with debugging, it is pointing to commit 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing") which introduced this regression. Disabling memcg causes the null pointer dereference at uninitialized data in function mem_cgroup_track_foreign_dirty_slowpath().
Fix it by returning directly if memcg is disabled, but not trying to record the foreign writebacks with dirty pages.
Fixed: 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing")
---v1->v2:
Move the checking code into mem_cgroup_track_foreign_dirty_slowpath() because usually memcg is enabled by default, and the original condition checking in mem_cgroup_track_foreign_dirty() is more unlikely being true to make it return.
mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+)
mm/memcontrol.c +3 -0
@@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,trace_track_foreign_dirty (page, wb); +if (mem_cgroup_disabled ()) +return ; +/* * Pick the slot to use. If there is already a slot for @wb, keep * using it. If not replace the oldest one which isn't being
Signature · 1 line
2.17.2
Michal Hocko
maintainermm/memcontrol.c void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, · line 4322
Quoted 11 lines · Baoquan He
mm/memcontrol.c +3
-0@@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,trace_track_foreign_dirty (page, wb); +if (mem_cgroup_disabled ()) +return ; +
This doesn't seem correct. We shouldn't even enter the slowpath with memcg disabled AFAIC. The check should be done at mem_cgroup_track_foreign_dirty level.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 9 lines · Michal Hocko
Quoted 8 lines · Baoquan He
mm/memcontrol.c +0
-0@@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,trace_track_foreign_dirty (page, wb);
Quoted 7 lines
mm/memcontrol.c void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, · line 4322
Quoted 3 lines
mm/memcontrol.c +3
-0+if (mem_cgroup_disabled ()) +return ; +This doesn't seem correct. We shouldn't even enter the slowpath with memcg disabled AFAIC. The check should be done at
mem_cgroup_track_foreign_dirtylevel.
You mean the way in v1 patch, right? It's also fine to me.
I am worried about the case that memcg is enabled, the checking by calling mem_cgroup_disabled() will lower efficiency. And it entering into mem_cgroup_track_foreign_dirty_slowpath() should be a rare event.
Thanks
Baoquan
Michal Hocko
maintainerQuoted 18 lines · Baoquan He
Quoted 9 lines · Michal Hocko
Quoted 8 lines · Baoquan He
mm/memcontrol.c +0
-0@@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,trace_track_foreign_dirty (page, wb);Quoted 7 lines
mm/memcontrol.c void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, · line 4322
Quoted 3 lines
mm/memcontrol.c +3
-0+if (mem_cgroup_disabled ()) +return ; +This doesn't seem correct. We shouldn't even enter the slowpath with memcg disabled AFAIC. The check should be done at
mem_cgroup_track_foreign_dirtylevel.
Quoted 4 lines
You mean the way in v1 patch, right? It's also fine to me.
I am worried about the case that memcg is enabled, the checking by calling
mem_cgroup_disabled()will lower efficiency.
This is hidden by a static branch so I wouldn't really be worried about the overhead.
Quoted 2 lines
And it entering
intomem_cgroup_track_foreign_dirty_slowpath()should be a rare event.
But &page->mem_cgroup->css != wb->memcg_css doesn't make any sense when memcg is disabled, right?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 24 lines · Michal Hocko
Quoted 18 lines · Baoquan He
Quoted 9 lines · Michal Hocko
Quoted 8 lines · Baoquan He
mm/memcontrol.c +0
-0@@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,trace_track_foreign_dirty (page, wb);Quoted 7 lines
mm/memcontrol.c void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, · line 4322
Quoted 3 lines
mm/memcontrol.c +3
-0+if (mem_cgroup_disabled ()) +return ; +This doesn't seem correct. We shouldn't even enter the slowpath with memcg disabled AFAIC. The check should be done at
mem_cgroup_track_foreign_dirtylevel.Quoted 4 lines
You mean the way in v1 patch, right? It's also fine to me.
I am worried about the case that memcg is enabled, the checking by calling
mem_cgroup_disabled()will lower efficiency.
Quoted 8 lines
This is hidden by a static branch so I wouldn't really be worried about the overhead.
Quoted 2 lines
And it entering
intomem_cgroup_track_foreign_dirty_slowpath()should be a rare event.But
&page->mem_cgroup->css!=wb->memcg_cssdoesn't make any sense when memcg is disabled, right?
Yeah, I think so. Make it like below?
@@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,
static inline void mem_cgroup_track_foreign_dirty(struct page *page,
struct bdi_writeback *wb)
{
- if (unlikely(&page->mem_cgroup->css != wb->memcg_css))
+ if (!mem_cgroup_disabled() &&
+ unlikely(&page->mem_cgroup->css != wb->memcg_css))
mem_cgroup_track_foreign_dirty_slowpath(page, wb);
}
Michal Hocko
maintainerQuoted 28 lines · Baoquan He
Quoted 27 lines · Michal Hocko
Quoted 18 lines · Baoquan He
Quoted 9 lines · Michal Hocko
Quoted 8 lines · Baoquan He
mm/memcontrol.c +0
-0@@ -4317,6 +4317,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,trace_track_foreign_dirty (page, wb);Quoted 7 lines
mm/memcontrol.c void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, · line 4322
Quoted 3 lines
mm/memcontrol.c +3
-0+if (mem_cgroup_disabled ()) +return ; +This doesn't seem correct. We shouldn't even enter the slowpath with memcg disabled AFAIC. The check should be done at
mem_cgroup_track_foreign_dirtylevel.Quoted 4 lines
You mean the way in v1 patch, right? It's also fine to me.
I am worried about the case that memcg is enabled, the checking by calling
mem_cgroup_disabled()will lower efficiency.This is hidden by a static branch so I wouldn't really be worried about the overhead.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
And it entering
intomem_cgroup_track_foreign_dirty_slowpath()should be a rare event.But
&page->mem_cgroup->css!=wb->memcg_cssdoesn't make any sense when memcg is disabled, right?Yeah, I think so. Make it like below?
Or just put it on its own line to make the code more readable.
Quoted 9 lines
(quoted diff) +2
-1@@ -1261,7 +1261,8 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page, static inline void mem_cgroup_track_foreign_dirty(struct page *page, struct bdi_writeback *wb) { - if (unlikely(&page->mem_cgroup->css != wb->memcg_css)) + if (!mem_cgroup_disabled() && + unlikely(&page->mem_cgroup->css != wb->memcg_css)) mem_cgroup_track_foreign_dirty_slowpath(page, wb); }
Signature · 2 lines
Michal Hocko SUSE Labs
PATCH v3 1/1 memcg: Only record foreign writebacks with dirty pages when memcg is not disabled
In reply to Baoquan HeIn kdump kernel, memcg usually is disabled with 'cgroup_disable=memory' for saving memory. Now kdump kernel will always panic when dump vmcore to local disk:
BUG: kernel NULL pointer dereference, address: 0000000000000ab8 PGD 5fcab067 P4D 5fcab067 PUD 5ff73067 PMD 0 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 598 Comm: makedumpfile Not tainted 5.3.0+ #26 Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 RIP: 0010:mem_cgroup_track_foreign_dirty_slowpath+0x38/0x140 Code: 55 48 8b 2d 6a bd 12 01 53 4c 8b 67 38 0f 1f 44 00 00 49 8b 06 48 89 e9 31 db be ff ff ff ff 48 8b 38 49 8d 84 24 d0 0a 00 00 <48> 39 78 e8 74 5a 48 8b 50 f8 48 39 ca 79 0e 44 8b 00 41 83 f8 01 RSP: 0018:ffffbc9300817bd0 EFLAGS: 00010046 RAX: 0000000000000ad0 RBX: 0000000000000000 RCX: 00000000fffba227 RDX: fffffffffffffff8 RSI: 00000000ffffffff RDI: 0000000000000004 RBP: 00000000fffba227 R08: 0000000000030340 R09: ffffbc9300817d10 R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000000 R13: ffff999c1fa2bcf0 R14: ffff999c1f7bcc78 R15: ffffe51fc177a8c0 FS: 00007f84f14c2b80(0000) GS:ffff999c22200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000ab8 CR3: 000000005f564000 CR4: 00000000003406b0 Call Trace: __set_page_dirty+0x52/0xc0 iomap_set_page_dirty+0x50/0x90 iomap_write_end+0x6e/0x270 iomap_write_actor+0xce/0x170 ? iomap_write_end+0x270/0x270 iomap_apply+0xba/0x11e ? iomap_write_end+0x270/0x270 iomap_file_buffered_write+0x62/0x90 ? iomap_write_end+0x270/0x270 xfs_file_buffered_aio_write+0xca/0x320 [xfs] new_sync_write+0x12d/0x1d0 vfs_write+0xa5/0x1a0 ksys_write+0x59/0xd0 do_syscall_64+0x59/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f84f1093ab5
And this will corrupt the 1st kernel too with 'cgroup_disable=memory'.
From the trace and with debugging, it is pointing to commit 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing") which introduced this regression. Disabling memcg causes the null pointer dereference at uninitialized data in function mem_cgroup_track_foreign_dirty_slowpath().
Fix it by returning directly if memcg is disabled, but not trying to record the foreign writebacks with dirty pages.
---v2->v3:
Move the checking into mem_cgroup_track_foreign_dirty() per Michal's suggestions.
include/linux/memcontrol.h | 3 +++ 1 file changed, 3 insertions(+)
include/linux/memcontrol.h +3 -0
@@ -1261,6 +1261,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,static inline void mem_cgroup_track_foreign_dirty (struct page *page,struct bdi_writeback *wb) { +if (mem_cgroup_disabled ()) +return ; +if (unlikely (&page->mem_cgroup->css != wb->memcg_css))mem_cgroup_track_foreign_dirty_slowpath (page, wb); }
Signature · 1 line
2.17.2
Quoted 46 lines · Baoquan He
In kdump kernel, memcg usually is disabled with 'cgroup_disable=memory' for saving memory. Now kdump kernel will always panic when dump vmcore to local disk:
BUG: kernel NULL pointer dereference, address: 0000000000000ab8 PGD 5fcab067 P4D 5fcab067 PUD 5ff73067 PMD 0 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 598 Comm: makedumpfile Not tainted 5.3.0+ #26 Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/02/2018 RIP: 0010:mem_cgroup_track_foreign_dirty_slowpath+0x38/0x140 Code: 55 48 8b 2d 6a bd 12 01 53 4c 8b 67 38 0f 1f 44 00 00 49 8b 06 48 89 e9 31 db be ff ff ff ff 48 8b 38 49 8d 84 24 d0 0a 00 00 <48> 39 78 e8 74 5a 48 8b 50 f8 48 39 ca 79 0e 44 8b 00 41 83 f8 01 RSP: 0018:ffffbc9300817bd0 EFLAGS: 00010046 RAX: 0000000000000ad0 RBX: 0000000000000000 RCX: 00000000fffba227 RDX: fffffffffffffff8 RSI: 00000000ffffffff RDI: 0000000000000004 RBP: 00000000fffba227 R08: 0000000000030340 R09: ffffbc9300817d10 R10: 0000000000000000 R11: 0000000000001000 R12: 0000000000000000 R13: ffff999c1fa2bcf0 R14: ffff999c1f7bcc78 R15: ffffe51fc177a8c0 FS: 00007f84f14c2b80(0000) GS:ffff999c22200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000ab8 CR3: 000000005f564000 CR4: 00000000003406b0 Call Trace: __set_page_dirty+0x52/0xc0 iomap_set_page_dirty+0x50/0x90 iomap_write_end+0x6e/0x270 iomap_write_actor+0xce/0x170 ? iomap_write_end+0x270/0x270 iomap_apply+0xba/0x11e ? iomap_write_end+0x270/0x270 iomap_file_buffered_write+0x62/0x90 ? iomap_write_end+0x270/0x270 xfs_file_buffered_aio_write+0xca/0x320 [xfs] new_sync_write+0x12d/0x1d0 vfs_write+0xa5/0x1a0 ksys_write+0x59/0xd0 do_syscall_64+0x59/0x1e0 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f84f1093ab5And this will corrupt the 1st kernel too with 'cgroup_disable=memory'.
Quoted 1 line
From the trace and with debugging, it is pointing to commit
97b27821b485("writeback, memcg: Implement foreign dirty flushing") which introduced this regression. Disabling memcg causes the null pointer dereference at uninitialized data in function
mem_cgroup_track_foreign_dirty_slowpath().
Quoted 5 lines
Fix it by returning directly if memcg is disabled, but not trying to record the foreign writebacks with dirty pages.
Fixes 97b27821b485 ("writeback, memcg: Implement foreign dirty flushing")Signed-off-by Baoquan He <[email protected]>
Looks good to me. This is essentially the same thing as if MEMCG was disabled completely AFAICS.
Thanks!
Quoted 24 lines
---
v2->v3:
Move the checking intomem_cgroup_track_foreign_dirty()per Michal's suggestions.include/linux/memcontrol.h | 3 +++ 1 file changed, 3 insertions(+)include/linux/memcontrol.h +3
-0@@ -1261,6 +1261,9 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct page *page,static inline void mem_cgroup_track_foreign_dirty (struct page *page,struct bdi_writeback *wb) { +if (mem_cgroup_disabled ()) +return ; +if (unlikely (&page->mem_cgroup->css != wb->memcg_css))mem_cgroup_track_foreign_dirty_slowpath (page, wb); }Signature · 1 line
2.17.2
Signature · 2 lines
Michal Hocko SUSE Labs