question on page-migration code

Hirokazu (and Marcelo),

In testing my manual page migration code, I've run up against a situation where the migrations are occasionally very slow. They work ok, but they can take minutes to migrate a few megabytes of memory.

Dropping into kdb shows that the migration code is waiting in msleep() in migrate_page_common() due to an -EAGAIN return from page_migratable(). A little further digging shows that the specific return in page_migratable() is the very last one there at the bottom of the routine.

I'm puzzled as to why the page is still busy in this case. Previous code in page_migratable() has unmapped the page, its not in PageWriteback() because we would have taken a different return statement in that case.

According to /proc/meminfo, there are no pages in either SwapCache or Dirty state, and the system has been sync'd before the migrate_pages() call was issued.

Quoted 11 lines · Ray Bryant

Hirokazu (and Marcelo),

In testing my manual page migration code, I've run up against a situation where the migrations are occasionally very slow. They work ok, but they can take minutes to migrate a few megabytes of memory.

Dropping into kdb shows that the migration code is waiting in msleep() in migrate_page_common() due to an -EAGAIN return from page_migratable(). A little further digging shows that the specific return in page_migratable() is the very last one there at the bottom of the routine.

Quoted 7 lines

I'm puzzled as to why the page is still busy in this case. Previous code in page_migratable() has unmapped the page, its not in PageWriteback() because we would have taken a different return statement in that case.

According to /proc/meminfo, there are no pages in either SwapCache or Dirty state, and the system has been sync'd before the migrate_pages() call was issued.

Who is using the page?

A little debugging might help similar to what bad_page does can help:

        printk(KERN_EMERG "flags:0x%0*lx mapping:%p mapcount:%d count:%d\n",
                (int)(2*sizeof(page_flags_t)), (unsigned long)page->flags,
                page->mapping, page_mapcount(page), page_count(page));

--

Quoted 11 lines · Ray Bryant

Hirokazu (and Marcelo),

In testing my manual page migration code, I've run up against a situation where the migrations are occasionally very slow. They work ok, but they can take minutes to migrate a few megabytes of memory.

Dropping into kdb shows that the migration code is waiting in msleep() in migrate_page_common() due to an -EAGAIN return from page_migratable(). A little further digging shows that the specific return in page_migratable() is the very last one there at the bottom of the routine.

Quoted 7 lines

I'm puzzled as to why the page is still busy in this case. Previous code in page_migratable() has unmapped the page, its not in PageWriteback() because we would have taken a different return statement in that case.

According to /proc/meminfo, there are no pages in either SwapCache or Dirty state, and the system has been sync'd before the migrate_pages() call was issued.

Who is using the page?

A little debugging might help similar to what bad_page does can help:

        printk(KERN_EMERG "flags:0x%0*lx mapping:%p mapcount:%d count:%d\n",
                (int)(2*sizeof(page_flags_t)), (unsigned long)page->flags,
                page->mapping, page_mapcount(page), page_count(page));

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 24 lines

Quoted 11 lines

Hirokazu (and Marcelo),

In testing my manual page migration code, I've run up against a situation where the migrations are occasionally very slow. They work ok, but they can take minutes to migrate a few megabytes of memory.

Dropping into kdb shows that the migration code is waiting in msleep() in migrate_page_common() due to an -EAGAIN return from page_migratable(). A little further digging shows that the specific return in page_migratable() is the very last one there at the bottom of the routine.

Quoted 7 lines

I'm puzzled as to why the page is still busy in this case. Previous code in page_migratable() has unmapped the page, its not in PageWriteback() because we would have taken a different return statement in that case.

According to /proc/meminfo, there are no pages in either SwapCache or Dirty state, and the system has been sync'd before the migrate_pages() call was issued.

Who is using the page?

A little debugging might help similar to what bad_page does can help:

Quoted 9 lines

        printk(KERN_EMERG "flags:0x%0*lx mapping:%p mapcount:%d count:%d\n",
                (int)(2*sizeof(page_flags_t)), (unsigned long)page->flags,
                page->mapping, page_mapcount(page), page_count(page));

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

A little further digging shows that when we go into -EAGAIN case in migrate_page_common(), we have flag bits 0x104D set, and when we finally exit the routine, we have flags bits 0x004D set. The 1 bit there is PG_private, as near as I can tell (not PG_arch_1, I guess I can't count).

PagePrivate() is cleared by truncation specific code in migrate_onepage(), but it doesn't appear to be cleared (directly) by code on the generic_migrate_page() patch. I wonder if this has something to do with the problem I am seeing.

Hi Ray,

I understand what happened on your machine.

PG_private is a filesystem specific flag, setting some filesystem depending data in page->private. When the flag is set on a page, only the local filesystem on which the page depends can handle it.

Most of the filesystems uses page->private to manage buffers while others may use it for different purposes. Each filesystem can implement migrate_page method to handles page->private. At this moment, only ext2 and ext3 have this method, which migrates buffers without any I/Os.

If the method isn't implemented for the page, the migration code calls pageout() and try_to_release_page() to release page->private instead.

Which filesystem are you using? I guess it might be XFS which doesn't have the method yet.

Thank you,
Hirokazu Takahashi.

What appears to be happening is the following:

dirty pte bits are being swept into the page dirty bit as a side effect of migration. That is, if a page had pte_dirty(pte) set, then after migration, it will have PageDirty(page) = true.

Only pages with PageDirty() set will be written to swap as part of the process of trying to clear PG_private. So, when I do the first migration, the PG_dirty bit is not set on the page, but the dirty bit is set in the pte. Because PG_dirty is not set, the page does not get written to swap, and the migration is fast. However, at the end of the migration process, the pages all have PG_dirty set and the pte dirty bits are cleared.

I guess it seems to me that if a page has pte dirty set, but doesn't have PG_dirty set, then that state should be carried over to the newpage after a migration, rather than sweeping the pte dirty bit into the PG_dirty bit.

Hi Hirokazu,

What appears to be happening is the following:

dirty pte bits are being swept into the page dirty bit as a side effect of migration. That is, if a page had pte_dirty(pte) set, then after migration, it will have PageDirty(page) = true.

Only pages with PageDirty() set will be written to swap as part of the process of trying to clear PG_private. So, when I do the first migration, the PG_dirty bit is not set on the page, but the dirty bit is set in the pte. Because PG_dirty is not set, the page does not get written to swap, and the migration is fast. However, at the end of the migration process, the pages all have PG_dirty set and the pte dirty bits are cleared.

The second time I do the migration, the PG_dirty bits are still set (left over from the first migration), so they have to be written to swap and the migration is slow. As part of the pageout(), try_to_release_page() process, the PG_dirty is cleared, along with the pte dirty bits, as before.

When the program is resumed, it will cause the pte dirty bits to be set, and then we will be back in the situation we started with before the first migration.

Hence the third migration will be fast, and the 4th migration will be slow, etc. This is a stable, repeatable process.

I guess it seems to me that if a page has pte dirty set, but doesn't have PG_dirty set, then that state should be carried over to the newpage after a migration, rather than sweeping the pte dirty bit into the PG_dirty bit.

Another way to do this would be to implement the migrate dirty buffers without swap I/O trick of ext2/3 in XFS, but that is somewhat far afield for me to try. :-) I'll discuss this with Nathan Scott et al and see if that is something that would be straightforward to do.

But I have a nagging suspicion that this covers up, rather than fixes the state transition from oldpage to newpage that really shouldn't be happening, as near as I can tell.

BTW, the program that I am testing creates a relatively large mapped file, and, as you guessed, this file is backed by XFS. Programs that just use large amounts of anonymous storage are not effected by this problem, I would imagine.

Signature · 14 lines

Best Regards,
Ray
-----------------------------------------------
                   Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
            so I installed Linux.
-----------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

If the method isn't implemented for the page, the migration code calls pageout() and try_to_release_page() to release page->private instead.

Which filesystem are you using? I guess it might be XFS which doesn't have the method yet.

Quoted 6 lines

If the method isn't implemented for the page, the migration code calls pageout() and try_to_release_page() to release page->private instead.

Which filesystem are you using? I guess it might be XFS which doesn't have the method yet.

Can we more easily detect and work around this in the code, so that this won't happen for more filesystems?

Hi,

Quoted 7 lines

Quoted 6 lines

If the method isn't implemented for the page, the migration code calls pageout() and try_to_release_page() to release page->private instead.

Which filesystem are you using? I guess it might be XFS which doesn't have the method yet.

Quoted 2 lines

Can we more easily detect and work around this in the code, so that this won't happen for more filesystems?

As Ray said, the following seems to be a straight approach. I haven't had any other ideas to work around it.

From my understanding there are two problems:

ray> I guess it seems to me that if a page has pte dirty set, but doesn't have ray> PG_dirty set, then that state should be carried over to the newpage after ray> a migration, rather than sweeping the pte dirty bit into the PG_dirty bit.

The implementation might be as follows:
- to make try_to_unmap_one() record dirty bit in anywhere instead of calling set_page_dirty().
- to make touch_unmapped_address() call get_user_pages() with the record of the dirty bit.

However, we have to remember that there must exit some race conditions. For example, it may fail to restore the dirty bit since the process address spaces might be deleted during the memory migration. This may occur as the process isn't suspended during the migration.

On Wed, Apr 13, 2005 at 07:48:00PM +0900, Hirokazu Takahashi wrote:

2) PG_dirty bit set on anonymous pages which have been migrated.

Quoted 3 lines

ray> I guess it seems to me that if a page has pte dirty set, but doesn't have ray> PG_dirty set, then that state should be carried over to the newpage after ray> a migration, rather than sweeping the pte dirty bit into the PG_dirty bit.

The dirty bit is set by swap allocation and freeing code.

Quoted 5 lines

The implementation might be as follows:
- to make try_to_unmap_one() record dirty bit in anywhere instead of calling set_page_dirty().
- to make touch_unmapped_address() call get_user_pages() with the record of the dirty bit.

Quoting Ray:
"Checking /proc/vmstat/pgpgout appears to indicate that the pages I am migrating are being swapped out when I see the migration slow down, although something is fishy with pgpgout."

Anonymous pages seem to the problem Ray is seeing, except (1) which vanishes with ext2/ext3 as he reports.

One point is that if free memory is below the safe watermarks, the system will vmscan, allocating swap & writing out, which is expected.

Quoted 4 lines

However, we have to remember that there must exit some race conditions. For example, it may fail to restore the dirty bit since the process address spaces might be deleted during the memory migration. This may occur as the process isn't suspended during the migration.

The PG_dirty bit is set, by the migration code, for anonymous pages only.

That said, I see no need to reset PG_dirty in case it was not set before migration, as you propose.

At Thu, 14 Apr 2005 12:57:34 -0300,

Quoted 22 lines · Marcelo Tosatti

On Wed, Apr 13, 2005 at 07:48:00PM +0900, Hirokazu Takahashi wrote:

2) PG_dirty bit set on anonymous pages which have been migrated.

Quoted 3 lines

ray> I guess it seems to me that if a page has pte dirty set, but doesn't have ray> PG_dirty set, then that state should be carried over to the newpage after ray> a migration, rather than sweeping the pte dirty bit into the PG_dirty bit.

The dirty bit is set by swap allocation and freeing code.

Quoted 5 lines

The implementation might be as follows:
- to make try_to_unmap_one() record dirty bit in anywhere instead of calling set_page_dirty().
- to make touch_unmapped_address() call get_user_pages() with the record of the dirty bit.

Quoting Ray:
"Checking /proc/vmstat/pgpgout appears to indicate that the pages I am migrating are being swapped out when I see the migration slow down, although something is fishy with pgpgout."

Quoted 2 lines

Anonymous pages seem to the problem Ray is seeing, except (1) which vanishes with ext2/ext3 as he reports.

I think Ray is using the word "swap" to mean "page out" and anonymous pages are irrelevant here, judging from his another mail (quoted below).

At Tue, 12 Apr 2005 00:43:42 -0500,
Ray Bryant wrote:
: BTW, the program that I am testing creates a relatively large mapped file, : and, as you guessed, this file is backed by XFS. Programs that just use : large amounts of anonymous storage are not effected by this problem, I : would imagine.

Quoted 2 lines

One point is that if free memory is below the safe watermarks, the system will vmscan, allocating swap & writing out, which is expected.

If there are enough RAM, mmaped dirty pages shouldn't be written back. However, memory migration triggers writebacks.

Quoted 5 lines

Quoted 4 lines

However, we have to remember that there must exit some race conditions. For example, it may fail to restore the dirty bit since the process address spaces might be deleted during the memory migration. This may occur as the process isn't suspended during the migration.

Quoted 1 line

The PG_dirty bit is set, by the migration code, for anonymous pages only.

If a file page is mmaped and its PTE is dirty, the page gets PG_dirty bit when it is unmapped.

Quoted 2 lines

That said, I see no need to reset PG_dirty in case it was not set before migration, as you propose.

I think PG_dirty should be reset, as the side effect is probably unacceptable for Ray's application. It would be a bit more complicated than just changing page and PTE bits, but I think it's doable.

Quoted 10 lines · IWAMOTO Toshihiro

At Thu, 14 Apr 2005 12:57:34 -0300,

Quoted 2 lines · Marcelo Tosatti

That said, I see no need to reset PG_dirty in case it was not set before migration, as you propose.

I think PG_dirty should be reset, as the side effect is probably unacceptable for Ray's application. It would be a bit more complicated than just changing page and PTE bits, but I think it's doable.

Yes, makes sense.

Question: Who is causing the writeouts here?

Is there memory pressure or is it pdflush?

Its not the migration code? (that would be a problem I think).

Hirokazu et al,

I'm sorry, I've been kind of out of the loop here since last Wenesday (that's the day I left Austin to fly to Melbourne, Australia which is where I am now, visiting the SGI lab in Melbourne).

Nathan Scott (who works at SGI Melbourne) looked at the ext2/ext3 migrate_page code and realized that basically the same implementation would work for xfs. So I now have a kernel that implements that function for xfs and, as you predicted, the "slow down" in the 2nd migration that I was seeing before has gone away. I'll add Nathan's patch to my manual page migration stuff in the next version (later this week, I hope).

So I guess it doesn't matter to me at the moment whether or not the PG_dirty bit is set on the pages, except that I philosphically dislike the fact that migration changes the state of the page. I'm not sure it matters, but I would prefer it if this didn't happen. However, I'm not adamant about this, since what I really want to happen is to have a functioning manual page migration system call. It does seem to be a bother to have to add that migrate_page method to each file system, since in most cases the addition is going to look somewhat like it does for ext2/3.

For xfs, Nathan did add an additional bit to make sure that xfs metadata pages were not considered migratable.

WRT, Marcelo's question as to who is causing the page out I/O to occur during migration, let me go back and verify this is actually what is happening.

Otherwise, is there a consensus about what to do about the PG_dirty bits being set on the migrated pages? As I read things Marcelo says it is not worth it, but others think that it should be fixed?

I understand what happened on your machine.

PG_private is a filesystem specific flag, setting some filesystem depending data in page->private. When the flag is set on a page, only the local filesystem on which the page depends can handle it.

Most of the filesystems uses page->private to manage buffers while others may use it for different purposes. Each filesystem can implement migrate_page method to handles page->private. At this moment, only ext2 and ext3 have this method, which migrates buffers without any I/Os.

If the method isn't implemented for the page, the migration code calls pageout() and try_to_release_page() to release page->private instead.

Which filesystem are you using? I guess it might be XFS which doesn't have the method yet.

Thank you,
Hirokazu Takahashi.

Who is using the page?

A little debugging might help similar to what bad_page does can help:

        printk(KERN_EMERG "flags:0x%0*lx mapping:%p mapcount:%d count:%d\n",
                (int)(2*sizeof(page_flags_t)), (unsigned long)page->flags,
                page->mapping, page_mapcount(page), page_count(page));

--