Offline Memory gets stuck in offline_pages()

Hi all

Overview:
During testing the CXL memory hotremove, we noticed that daxctl offline-memory dax0.0 would get stuck forever sometimes. daxctl offline-memory dax0.0 will write "offline" to /sys/devices/system/memory/memoryNNN/state.

Workaround:
When it happens, we can type Ctrl-C to abort it and then retry again. Then the CXL memory is able to offline successfully.

Where the kernel gets stuck:
After digging into the kernel, we found that when the issue occurs, the kernel is stuck in the outer loop of offline_pages(). Below is a piece of the highlighted offline_pages():

```
int __ref offline_pages()
{
    do { // outer loop
      pfn = start_pfn;
      do {
        ret = scan_movable_pages(pfn, end_pfn, &pfn);  // It returns -ENOENT
        if (!ret)
           do_migrate_range(pfn, end_pfn);             // Not reach here
      } while (!ret);
      ret = test_pages_isolated(start_pfn, end_pfn, MEMORY_OFFLINE);
      } while (ret);                                   // ret is -EBUSY
}
```

In this case, we dumped the first page that cannot be isolated (see dump_page below), it's content does not change in each iteration.:
```

Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:29:26 linux kernel: page dumped because: trouble page...

```

Every time the issue occurs, the content of the page structure is similar.

Questions:
Q1. Is this behavior expected? At least for an OS administrator, it should return
promptly (success or failure) instead of hanging indefinitely.

Q2. Regarding the offline_pages() function, encountering such a page indeed causes
an endless loop. Shouldn't another part of the kernel timely changed the state of this page?

When I use the workaround mentioned above (Ctrl-C and try offline again), I find that the page state changes (see dump_page below):

```
Jun 28 15:33:12 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:33:12 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:33:12 linux kernel: raw: 009fffffc0000000 dead000000000100 dead000000000122 0000000000000000
Jun 28 15:33:12 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:33:12 linux kernel: page dumped because: previous trouble page
```

What our test does:
We have a CXL memory device, which is configured as kmem and online into the MOVABLE zone as NUMA node2. We run two processes, consume-memory and offline-memory, in parallel, see the pseudo code below:

```
main()
{
if (fork() == 0)
numactl -m 2 ./consume-memory

Quoted 5 lines · Zhijian Li (Fujitsu)

Hi all
Overview:
During testing the CXL memory hotremove, we noticed that daxctl offline-memory dax0.0 would get stuck forever sometimes. daxctl offline-memory dax0.0 will write "offline" to /sys/devices/system/memory/memoryNNN/state.

Hi,
See
Documentation/admin-guide/mm/memory-hotplug.rst

"
Further, when running into out of memory situations while migrating pages, or when still encountering permanently unmovable pages within ZONE_MOVABLE (-> BUG), memory offlining will keep retrying until it eventually succeeds.
When offlining is triggered from user space, the offlining context can be terminated by sending a signal. A timeout based offlining can easily be implemented via::
    % timeout $TIMEOUT offline_block | failure_handling
"

Quoted 21 lines

Workaround:
When it happens, we can type Ctrl-C to abort it and then retry again. Then the CXL memory is able to offline successfully.
Where the kernel gets stuck:
After digging into the kernel, we found that when the issue occurs, the kernel is stuck in the outer loop of offline_pages(). Below is a piece of the highlighted offline_pages():

```
int __ref offline_pages()
{
     do { // outer loop
       pfn = start_pfn;
       do {
         ret = scan_movable_pages(pfn, end_pfn, &pfn);  // It returns -ENOENT
         if (!ret)
            do_migrate_range(pfn, end_pfn);             // Not reach here
       } while (!ret);
       ret = test_pages_isolated(start_pfn, end_pfn, MEMORY_OFFLINE);
       } while (ret);                                   // ret is -EBUSY
}
```

Quoted 8 lines

In this case, we dumped the first page that cannot be isolated (see dump_page below), it's content does not change in each iteration.:
```

Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:29:26 linux kernel: page dumped because: trouble page...

Are you sure that's the problematic page?

refcount:0
Indicates that the page is free. But maybe it does not have PageBuddy() set. It could also be that this is a "tail" page of a PageBuddy() page,

and
somehow we always end up on the tail in test_pages_isolated(). Which kernel + architecture are you testing with?

Quoted 10 lines

```

Every time the issue occurs, the content of the page structure is similar.

Questions:
Q1. Is this behavior expected? At least for an OS administrator, it should return
      promptly (success or failure) instead of hanging indefinitely.

It's expected that it might take a long time (possibly forever) in corner cases. See documentation.

But it's likely unexpected that we have some problematic page here.

Quoted 5 lines

Q2. Regarding the offline_pages() function, encountering such a page indeed causes
      an endless loop. Shouldn't another part of the kernel timely changed the state
      of this page?

There are various things that can go wrong. One issue might be that we try migrating a page but continuously fail to allocate memory to be used as a migration target. It seems unlikely with the page you dumped above, though.

Do you maybe have that CXL memory be on a separate "fake" NUMA node,

and your workload mbind() itself to that NUMA node, possibly refusing to migrate somewhere else?

Quoted 24 lines

      When I use the workaround mentioned above (Ctrl-C and try offline again), I find
      that the page state changes (see dump_page below):

```
Jun 28 15:33:12 linux kernel: page: refcount:0 mapcount:0 
mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:33:12 linux kernel: flags: 
0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:33:12 linux kernel: raw: 009fffffc0000000 dead000000000100 
dead000000000122 0000000000000000
Jun 28 15:33:12 linux kernel: raw: 0000000000000000 0000000000000000 
00000000ffffffff 0000000000000000
Jun 28 15:33:12 linux kernel: page dumped because: previous trouble page
```

What our test does:
We have a CXL memory device, which is configured as kmem and online into the MOVABLE
zone as NUMA node2. We run two processes, consume-memory and offline-memory, in parallel,
see the pseudo code below:

Quoted 5 lines

```
main()
{
      if (fork() == 0)
          numactl -m 2 ./consume-memory

What exactly does "consume-memory" do? Does it involve hugetlb maybe?

Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:29:26 linux kernel: page dumped because: trouble page...

Hi all

Overview:
During testing the CXL memory hotremove, we noticed that daxctl offline-memory dax0.0 would get stuck forever sometimes. daxctl offline-memory dax0.0 will write "offline" to /sys/devices/system/memory/memoryNNN/state.

Workaround:
When it happens, we can type Ctrl-C to abort it and then retry again. Then the CXL memory is able to offline successfully.

Where the kernel gets stuck:
After digging into the kernel, we found that when the issue occurs, the kernel is stuck in the outer loop of offline_pages(). Below is a piece of the highlighted offline_pages():

```
int __ref offline_pages()
{
    do { // outer loop
      pfn = start_pfn;
      do {
        ret = scan_movable_pages(pfn, end_pfn, &pfn);  // It returns -ENOENT
        if (!ret)
           do_migrate_range(pfn, end_pfn);             // Not reach here
      } while (!ret);
      ret = test_pages_isolated(start_pfn, end_pfn, MEMORY_OFFLINE);
      } while (ret);                                   // ret is -EBUSY
}
```

In this case, we dumped the first page that cannot be isolated (see dump_page below), it's content does not change in each iteration.:

```
Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:29:26 linux kernel: page dumped because: trouble page...
```

Every time the issue occurs, the content of the page structure is similar.

Questions:
Q1. Is this behavior expected? At least for an OS administrator, it should return
promptly (success or failure) instead of hanging indefinitely.
Q2. Regarding the offline_pages() function, encountering such a page indeed causes
an endless loop. Shouldn't another part of the kernel timely changed the state of this page?

When I use the workaround mentioned above (Ctrl-C and try offline again), I find that the page state changes (see dump_page below):

```
Jun 28 15:33:12 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:33:12 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:33:12 linux kernel: raw: 009fffffc0000000 dead000000000100 dead000000000122 0000000000000000
Jun 28 15:33:12 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:33:12 linux kernel: page dumped because: previous trouble page
```

What our test does:
We have a CXL memory device, which is configured as kmem and online into the MOVABLE zone as NUMA node2. We run two processes, consume-memory and offline-memory, in parallel, see the pseudo code below:

```
main()
{
      if (fork() == 0)
          numactl -m 2 ./consume-memory
      else {
          daxctl offline-memory dax0.0
          wait()
      }
}
```

Attached is the process information (when it gets stuck):

```
root 25716 0.0 0.0 2460 1408 pts/0 S+ 15:28 0:00 ./main
root 25719 0.0 0.0 0 0 pts/0 Z+ 15:28 0:00 [consume-memory] <defunct>
root 25720 98.6 0.0 9476 3740 pts/0 R+ 15:28 0:26 daxctl offline-memory /dev/dax0.0
```

Feel free to let me know if you need more details.
Thank you for your attention to this issue. Looking forward to your insights.

Thanks
Zhijian

All,

Some progress updates

When issue occurs, calling __drain_all_pages() can make offline_pages() escape from the loop.

Quoted 7 lines

Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7980dd
Jun 28 15:29:26 linux kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788 ffffd4f0ffd97ef0 0000000000000000
Jun 28 15:29:26 linux kernel: raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000
Jun 28 15:29:26 linux kernel: page dumped because: trouble page...

With this problematic page structure contents, it seems that the list_head = {ffffdfbd9e603788, ffffd4f0ffd97ef0} is valid.

I guess it was linking to the pcp_list, so I dumped the per_cpu_pages[cpu].count in every in critical timings.

An example is as below,
offline_pages()

{
	// per_cpu_pages[1].count = 0
	zone_pcp_disable() // will call __drain_all_pages()
	// per_cpu_pages[1].count = 188
	do {
		do {
			scan_movable_pages()
			ret = do_migrate_range()
		} while (!ret)

		ret = test_pages_isolated()

		if(is the 1st iteration)
			// per_cpu_pages[1].count = 182

		if (issue occurs) { /* if the loop take beyond 10 seconds */
			// per_cpu_pages[1].count = 61
			__drain_all_pages()
			// per_cpu_pages[1].count = 0
			/* will escape from the outer loop in later iterations */
		}
	} while (ret)
}

Some interesting points:
- After the 1st __drain_all_pages(), per_cpu_pages[1].count increased to 188 from 0, does it mean it's racing with something...?
- per_cpu_pages[1].count will decrease but not decrease to 0 during iterations
- when issue occurs, calling __drain_all_pages() will decrease per_cpu_pages[1].count to 0.

So I wonder if it's fine to call __drain_all_pages() in the loop?

Looking forward to your insights.

-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Thursday, July 4, 2024 4:15 PM

Quoted 5 lines · Zhijian Li (Fujitsu)

All,

Some progress updates

When issue occurs, calling __drain_all_pages() can make offline_pages() escape

from the loop.

Quoted 8 lines

Quoted 8 lines

Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x7980dd Jun 28 15:29:26 linux
kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788

ffffd4f0ffd97ef0 0000000000000000 Jun 28 15:29:26 linux kernel: raw:
0000000000000000 0000000000000000 00000000ffffffff 0000000000000000

Jun 28 15:29:26 linux kernel: page dumped because: trouble page...

Quoted 2 lines

Quoted 1 line

Quoted 5 lines

With this problematic page structure contents, it seems that the list_head = {ffffdfbd9e603788, ffffd4f0ffd97ef0} is valid.

I guess it was linking to the pcp_list, so I dumped the per_cpu_pages[cpu].count in every in critical timings.

So, is your reproducer getting fixed when you call __drain_all_pages() in the loop? (not that it's the right fix, but a could datapoint :) )

Quoted 28 lines

An example is as below,
offline_pages()

{
	// per_cpu_pages[1].count = 0
	zone_pcp_disable() // will call __drain_all_pages()
	// per_cpu_pages[1].count = 188
	do {
		do {
			scan_movable_pages()
			ret = do_migrate_range()
		} while (!ret)

		ret = test_pages_isolated()

		if(is the 1st iteration)
			// per_cpu_pages[1].count = 182

		if (issue occurs) { /* if the loop take beyond 10 seconds */
			// per_cpu_pages[1].count = 61
			__drain_all_pages()
			// per_cpu_pages[1].count = 0
			/* will escape from the outer loop in later iterations */
		}
	} while (ret)
}

Quoted 2 lines

Some interesting points:
- After the 1st __drain_all_pages(), per_cpu_pages[1].count increased to 188

from 0,

Quoted 2 lines

does it mean it's racing with something...?
- per_cpu_pages[1].count will decrease but not decrease to 0 during

iterations

Quoted 1 line

- when issue occurs, calling __drain_all_pages() will decrease

per_cpu_pages[1].count to 0.

That's indeed weird. Maybe there is a race, or zone_pcp_disable() is not fully effective for a zone?

Quoted 5 lines

So I wonder if it's fine to call __drain_all_pages() in the loop?

Looking forward to your insights.

So, in free_unref_page(), we make sure to never place MIGRATE_ISOLATE onto the PCP. All pageblocks we are going to offline should be isolated at this point, so no page that is getting freed and part of the to-be-offlined range should end up on the PCP. So far the theory.

In offlining code we do

1) Set MIGRATE_ISOLATE
2) zone_pcp_disable() -> set high-and-batch to 0 and drain

Could there be a race in free_unref_page(), such that although zone_pcp_disable() succeeds, we would still end up with a page in the pcp? (especially, one that has MIGRATE_ISOLATE set for its pageblock?)

--
Cheers,

David / dhildenb

Quoted 26 lines

-----Original Message-----
From: David Hildenbrand <[email protected]>
Sent: Thursday, July 4, 2024 4:15 PM

Quoted 5 lines · Zhijian Li (Fujitsu)

All,

Some progress updates

When issue occurs, calling __drain_all_pages() can make offline_pages() escape

from the loop.

Quoted 8 lines

Quoted 8 lines

Jun 28 15:29:26 linux kernel: page: refcount:0 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x7980dd Jun 28 15:29:26 linux
kernel: flags: 0x9fffffc0000000(node=2|zone=3|lastcpupid=0x1fffff)
Jun 28 15:29:26 linux kernel: raw: 009fffffc0000000 ffffdfbd9e603788

ffffd4f0ffd97ef0 0000000000000000 Jun 28 15:29:26 linux kernel: raw:
0000000000000000 0000000000000000 00000000ffffffff 0000000000000000

Jun 28 15:29:26 linux kernel: page dumped because: trouble page...

Quoted 2 lines

Quoted 1 line

Quoted 8 lines

Quoted 5 lines

With this problematic page structure contents, it seems that the list_head = {ffffdfbd9e603788, ffffd4f0ffd97ef0} is valid.

I guess it was linking to the pcp_list, so I dumped the per_cpu_pages[cpu].count in every in critical timings.

So, is your reproducer getting fixed when you call __drain_all_pages() in the loop? (not that it's the right fix, but a could datapoint :) )

Yeah, it works for my reproducer.

Quoted 31 lines

Quoted 28 lines

An example is as below,
offline_pages()

{
	// per_cpu_pages[1].count = 0
	zone_pcp_disable() // will call __drain_all_pages()
	// per_cpu_pages[1].count = 188
	do {
		do {
			scan_movable_pages()
			ret = do_migrate_range()
		} while (!ret)

		ret = test_pages_isolated()

		if(is the 1st iteration)
			// per_cpu_pages[1].count = 182

		if (issue occurs) { /* if the loop take beyond 10 seconds */
			// per_cpu_pages[1].count = 61
			__drain_all_pages()
			// per_cpu_pages[1].count = 0
			/* will escape from the outer loop in later iterations */
		}
	} while (ret)
}

Quoted 2 lines

Some interesting points:
- After the 1st __drain_all_pages(), per_cpu_pages[1].count increased to 188

from 0,

Quoted 8 lines

Quoted 2 lines

does it mean it's racing with something...?
- per_cpu_pages[1].count will decrease but not decrease to 0 during

iterations

Quoted 1 line

- when issue occurs, calling __drain_all_pages() will decrease

per_cpu_pages[1].count to 0.

That's indeed weird. Maybe there is a race, or zone_pcp_disable() is not fully effective for a zone?

I often see there still are pages in PCP after the zone_pcp_disable().

Quoted 14 lines

Quoted 5 lines

So I wonder if it's fine to call __drain_all_pages() in the loop?

Looking forward to your insights.

So, in free_unref_page(), we make sure to never place MIGRATE_ISOLATE onto the PCP. All pageblocks we are going to offline should be isolated at this point, so no page that is getting freed and part of the to-be-offlined range should end up on the PCP. So far the theory.

In offlining code we do

Quoted 6 lines

1) Set MIGRATE_ISOLATE
2) zone_pcp_disable() -> set high-and-batch to 0 and drain

Could there be a race in free_unref_page(), such that although zone_pcp_disable() succeeds, we would still end up with a page in the pcp? (especially, one that has MIGRATE_ISOLATE set for its pageblock?)

Thanks for your idea, I will further investigate in this direction.

Thanks
Zhijian

Quoted 6 lines

--
Cheers,

David / dhildenb

          CPU0                                  CPU1
      -----------                           ---------

// erase pcp_list
zone_pcp_disable // pcp->count = 0

lru_cache_diable() __rmqueue_pcplist() // re-add pages to pcp_lsit
__rmqueue_pcplist() // drop pages from pcp_list
decay_pcp_high() // drop pages from pcp_list
loop { ...
__rmqueue_pcplist() // drop pages from pcp_list,
// it will be only called a few times during the loop
scan_movable_pages() ...
migration_pages() decay_pcp_high() // drop pages from pcp_list, it will be called by
// a worker periodically during the loop

// wait pcp_list to be empty
} while (test_pages_isolated())

And we noticed that re-add pages to pcp_list in '__rmqueue_pcplist()` only happen once, pcp->count changed to 200 from 0 for example.

The later calls to __rmqueue_pcplist() will drop pcp->count by step 1 for each call, For example, pcp->count: 199->198->197->196...,
However it stops calling __rmqueue_pcplist after a few times before pcp->count is dropped to 0.

In the normal/good case, we also noticed that __rmqueue_pcplist() dropped pcp->count to 0.