Skip to contentSkip to conversation

Regression: x86/mm: new _PTE_SWP_SOFT_DIRTY bit conflicts with existing use

DISCUSSION David Vrabel 13 participants 88 messages Aug 21, 2013 – Aug 27, 2013 x86

lore
Share & tools
lore
#0 lore raw

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

The use of this bit also appears to preclude the use of (transparent) huge pages by the application. It is not clear if there is something else guaranteeing that that there will be no huge pages.

To fix this regression I suggest one or more of:

1. If no other changes are made, at a mimimum, MEM_SOFT_DIRTY must require !XEN and possibly !TRANSPARENT_HUGEPAGE and !HUGETLBFS. This would prevent this option being enabled on the majority of standard Linux distributions.

2. Find a different PTE bit to (re)use.

3. Avoid clearing the soft dirty bit when repopulating a swapped out page.

4. Redesign the soft dirty tracking to not require the use of architecture specific PTE bits. e.g., by using a shadow set of structures for the soft dirty bit tracking.

David

#1 lore raw
Quoted 10 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

Especially with filesystems which would end up using those pages (as the memory allocator would recycle them) and with corruption in the filesystem. Took months to figure that out.

Quoted 12 lines

The use of this bit also appears to preclude the use of (transparent) huge pages by the application. It is not clear if there is something else guaranteeing that that there will be no huge pages.

To fix this regression I suggest one or more of:

1. If no other changes are made, at a mimimum, MEM_SOFT_DIRTY must require !XEN and possibly !TRANSPARENT_HUGEPAGE and !HUGETLBFS. This would prevent this option being enabled on the majority of standard Linux distributions.

Quoted 7 lines

2. Find a different PTE bit to (re)use.

3. Avoid clearing the soft dirty bit when repopulating a swapped out page.

4. Redesign the soft dirty tracking to not require the use of architecture specific PTE bits. e.g., by using a shadow set of structures for the soft dirty bit tracking.

Or revert this patch and in 3.12 fix it using one of the options above or other ones.

Quoted 2 lines

David

#2 lore raw
In reply to David Vrabel
Quoted 10 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

Especially with filesystems which would end up using those pages (as the memory allocator would recycle them) and with corruption in the filesystem. Took months to figure that out.

Quoted 12 lines

The use of this bit also appears to preclude the use of (transparent) huge pages by the application. It is not clear if there is something else guaranteeing that that there will be no huge pages.

To fix this regression I suggest one or more of:

1. If no other changes are made, at a mimimum, MEM_SOFT_DIRTY must require !XEN and possibly !TRANSPARENT_HUGEPAGE and !HUGETLBFS. This would prevent this option being enabled on the majority of standard Linux distributions.

Quoted 7 lines

2. Find a different PTE bit to (re)use.

3. Avoid clearing the soft dirty bit when repopulating a swapped out page.

4. Redesign the soft dirty tracking to not require the use of architecture specific PTE bits. e.g., by using a shadow set of structures for the soft dirty bit tracking.

Or revert this patch and in 3.12 fix it using one of the options above or other ones.

Quoted 2 lines

David

#3 lore raw
In reply to David Vrabel

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Quoted 33 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT
table with WC so a page that was previously WB will end up as WC.

The use of this bit also appears to preclude the use of (transparent) huge pages by the application. It is not clear if there is something else guaranteeing that that there will be no huge pages.

To fix this regression I suggest one or more of:

1. If no other changes are made, at a mimimum, MEM_SOFT_DIRTY must require !XEN and possibly !TRANSPARENT_HUGEPAGE and !HUGETLBFS. This would prevent this option being enabled on the majority of standard Linux distributions.

2. Find a different PTE bit to (re)use.

3. Avoid clearing the soft dirty bit when repopulating a swapped out page.

4. Redesign the soft dirty tracking to not require the use of architecture specific PTE bits. e.g., by using a shadow set of structures for the soft dirty bit tracking.

David

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#4 lore raw
In reply to David Vrabel

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Quoted 33 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT
table with WC so a page that was previously WB will end up as WC.

The use of this bit also appears to preclude the use of (transparent) huge pages by the application. It is not clear if there is something else guaranteeing that that there will be no huge pages.

To fix this regression I suggest one or more of:

1. If no other changes are made, at a mimimum, MEM_SOFT_DIRTY must require !XEN and possibly !TRANSPARENT_HUGEPAGE and !HUGETLBFS. This would prevent this option being enabled on the majority of standard Linux distributions.

2. Find a different PTE bit to (re)use.

3. Avoid clearing the soft dirty bit when repopulating a swapped out page.

4. Redesign the soft dirty tracking to not require the use of architecture specific PTE bits. e.g., by using a shadow set of structures for the soft dirty bit tracking.

David

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#27 lore raw
In reply to David Vrabel
Quoted 11 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

#28 lore raw
In reply to David Vrabel
Quoted 11 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

#5 lore raw
In reply to H. Peter Anvin
Quoted 3 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap entry but always mark page read from swap as dirty one (it's better than having no tracker at all and will be the case on old machines with 32bit ptes only).

#6 lore raw
In reply to H. Peter Anvin
Quoted 3 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap entry but always mark page read from swap as dirty one (it's better than having no tracker at all and will be the case on old machines with 32bit ptes only).

#7 lore raw

OK now I'm confused. I guess I shouldn't comment while on vacation and cache cold on everything.

Quoted 13 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on

!Xen but we

Quoted 1 line

have gotten requests for WT support which would mean adding in the

PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap
entry but always mark page read from swap as dirty one (it's better than having
no tracker at all and will be the case on old machines with 32bit ptes only).

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#29 lore raw
In reply to Cyrill Gorcunov

Good question...

Quoted 17 lines · Cyrill Gorcunov
Quoted 4 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as

_PTE_PSE

Quoted 5 lines

and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the

PAT

Quoted 2 lines

table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#8 lore raw
In reply to Cyrill Gorcunov

OK now I'm confused. I guess I shouldn't comment while on vacation and cache cold on everything.

Quoted 13 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on

!Xen but we

Quoted 1 line

have gotten requests for WT support which would mean adding in the

PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap
entry but always mark page read from swap as dirty one (it's better than having
no tracker at all and will be the case on old machines with 32bit ptes only).

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#30 lore raw
In reply to Cyrill Gorcunov

Good question...

Quoted 17 lines · Cyrill Gorcunov
Quoted 4 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as

_PTE_PSE

Quoted 5 lines

and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the

PAT

Quoted 2 lines

table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#9 lore raw
In reply to H. Peter Anvin
Quoted 2 lines · H. Peter Anvin

OK now I'm confused. I guess I shouldn't comment while on vacation and cache cold on everything.

I rather think I'm missing something, that's why I asked David how this featue affects non present pte.

#10 lore raw
In reply to H. Peter Anvin
Quoted 2 lines · H. Peter Anvin

OK now I'm confused. I guess I shouldn't comment while on vacation and cache cold on everything.

I rather think I'm missing something, that's why I asked David how this featue affects non present pte.

#31 lore raw
In reply to Cyrill Gorcunov
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 6 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

Quoted 8 lines
Quoted 5 lines

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Jan

#32 lore raw
In reply to Cyrill Gorcunov
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 6 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

Quoted 8 lines
Quoted 5 lines

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Jan

#33 lore raw

Only WB pages should be swappable, but even so, the cacheability should be in the vma.

Quoted 39 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 3 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a

new

Quoted 1 line
Quoted 1 line

PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as

_PTE_PSE

Quoted 5 lines
Quoted 5 lines

and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the

PAT

Quoted 5 lines
Quoted 2 lines

table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Jan

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#34 lore raw
In reply to Jan Beulich

Only WB pages should be swappable, but even so, the cacheability should be in the vma.

Quoted 39 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 3 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a

new

Quoted 1 line
Quoted 1 line

PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as

_PTE_PSE

Quoted 5 lines
Quoted 5 lines

and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the

PAT

Quoted 5 lines
Quoted 2 lines

table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Jan

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#35 lore raw
In reply to Jan Beulich
Quoted 29 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 6 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

Quoted 8 lines
Quoted 5 lines

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Quoted 5 lines

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Only to non-present ptes, as far as I know.

do_swap_page

	...
	pte = mk_pte(page, vma->vm_page_prot);

	/* new pte from vm_page_prot generated */
	...
	set_pte_at(mm, address, page_table, pte);
	/* and assigned to old place */

with soft dirty in swap it is somehow more weirdy

	pte = mk_pte(page, vma->vm_page_prot);
	...
	if (pte_swp_soft_dirty(orig_pte))
		pte = pte_mksoft_dirty(pte);
	set_pte_at(mm, address, page_table, pte);

orig_pte has pse bit set if page has been soft dirty when it reached swap.

#36 lore raw
In reply to Jan Beulich
Quoted 29 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 6 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

Quoted 8 lines
Quoted 5 lines

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Quoted 5 lines

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Only to non-present ptes, as far as I know.

do_swap_page

	...
	pte = mk_pte(page, vma->vm_page_prot);

	/* new pte from vm_page_prot generated */
	...
	set_pte_at(mm, address, page_table, pte);
	/* and assigned to old place */

with soft dirty in swap it is somehow more weirdy

	pte = mk_pte(page, vma->vm_page_prot);
	...
	if (pte_swp_soft_dirty(orig_pte))
		pte = pte_mksoft_dirty(pte);
	set_pte_at(mm, address, page_table, pte);

orig_pte has pse bit set if page has been soft dirty when it reached swap.

#37 lore raw
Quoted 18 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 16 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Quoted 7 lines
Quoted 5 lines

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Only to non-present ptes, as far as I know.

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Quoted 11 lines

do_swap_page

	...
	pte = mk_pte(page, vma->vm_page_prot);

	/* new pte from vm_page_prot generated */
	...
	set_pte_at(mm, address, page_table, pte);
	/* and assigned to old place */

with soft dirty in swap it is somehow more weirdy

Quoted 8 lines
	pte = mk_pte(page, vma->vm_page_prot);
	...
	if (pte_swp_soft_dirty(orig_pte))
		pte = pte_mksoft_dirty(pte);
	set_pte_at(mm, address, page_table, pte);

orig_pte has pse bit set if page has been soft dirty when it reached swap.

"when it reached swap" to me again implies that it could come from a live page table, with the present bit set. So that explanation attempt of yours confuses me more than it clarifies things for me. (And referring to this bit as PSE bit is sort of wrong here too - there's no PSE bit for 4k PTEs, that bit is the PAT one, and that's what the whole discussion started from.)

Jan

#38 lore raw
In reply to Cyrill Gorcunov
Quoted 18 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 16 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

David, could you please explain, Xen keeps and analyze _PTE_PAT bit for ptes which are not present?

No, the problem isn't with not-present PTEs (i.e. swap entries), but with present ones - the same bit (7) is being used for both, according to this comment:

/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
* with swap entry format. On x86 bits 6 and 7 are *not* involved
* into swap entry computation, but bit 6 is used for nonlinear
* file mapping, so we borrow bit 7 for soft dirty tracking. */

Quoted 7 lines
Quoted 5 lines

Or are you telling me that the comment is misleading (at least me), and this applies only to not-present PTEs? And even then - where would the value of the original PAT bit be stored while swapped out (or is it impossible - now and forever - for WC pages to get swapped)?

Only to non-present ptes, as far as I know.

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Quoted 11 lines

do_swap_page

	...
	pte = mk_pte(page, vma->vm_page_prot);

	/* new pte from vm_page_prot generated */
	...
	set_pte_at(mm, address, page_table, pte);
	/* and assigned to old place */

with soft dirty in swap it is somehow more weirdy

Quoted 8 lines
	pte = mk_pte(page, vma->vm_page_prot);
	...
	if (pte_swp_soft_dirty(orig_pte))
		pte = pte_mksoft_dirty(pte);
	set_pte_at(mm, address, page_table, pte);

orig_pte has pse bit set if page has been soft dirty when it reached swap.

"when it reached swap" to me again implies that it could come from a live page table, with the present bit set. So that explanation attempt of yours confuses me more than it clarifies things for me. (And referring to this bit as PSE bit is sort of wrong here too - there's no PSE bit for 4k PTEs, that bit is the PAT one, and that's what the whole discussion started from.)

Jan

#39 lore raw
Quoted 5 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Quoted 7 lines
Quoted 2 lines

orig_pte has pse bit set if page has been soft dirty when it reached swap.

"when it reached swap" to me again implies that it could come from a live page table, with the present bit set. So that explanation attempt of yours confuses me more than it clarifies things for me. (And referring to this bit as PSE bit is

When page swapped out it become non-present in pte entry.

Quoted 3 lines

sort of wrong here too - there's no PSE bit for 4k PTEs, that bit is the PAT one, and that's what the whole discussion started from.)

And I asked David to point me how it happens, because I don't understand at which point pse bit get analized when page is not present.

#40 lore raw
In reply to Jan Beulich
Quoted 5 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Quoted 7 lines
Quoted 2 lines

orig_pte has pse bit set if page has been soft dirty when it reached swap.

"when it reached swap" to me again implies that it could come from a live page table, with the present bit set. So that explanation attempt of yours confuses me more than it clarifies things for me. (And referring to this bit as PSE bit is

When page swapped out it become non-present in pte entry.

Quoted 3 lines

sort of wrong here too - there's no PSE bit for 4k PTEs, that bit is the PAT one, and that's what the whole discussion started from.)

And I asked David to point me how it happens, because I don't understand at which point pse bit get analized when page is not present.

#11 lore raw
In reply to Cyrill Gorcunov
Quoted 8 lines · Cyrill Gorcunov
Quoted 3 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap entry but always mark page read from swap as dirty one (it's better than having no tracker at all and will be the case on old machines with 32bit ptes only).

Quite frankly, unless I see a patch later today that is

(a) obvious
(b) explains what is going on
(c) tested

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

Linus

#12 lore raw
In reply to Cyrill Gorcunov
Quoted 8 lines · Cyrill Gorcunov
Quoted 3 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap entry but always mark page read from swap as dirty one (it's better than having no tracker at all and will be the case on old machines with 32bit ptes only).

Quite frankly, unless I see a patch later today that is

(a) obvious
(b) explains what is going on
(c) tested

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

Linus

#13 lore raw
Quoted 11 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov
Quoted 3 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap entry but always mark page read from swap as dirty one (it's better than having no tracker at all and will be the case on old machines with 32bit ptes only).

Quite frankly, unless I see a patch later today that is

I'll do my best to fix it, but I've not yet been shown where the problem happens. Neither I'm convinced if it is here at all. I'm asking David for details.

Quoted 9 lines

(a) obvious
(b) explains what is going on
(c) tested

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess

This was not my invention on such encoding, I simply had no choise but follow existing scheme. Still there are cleanup patches in -mm tree which make code a way more readable (even former one, without soft-dirty bit).

Quoted 2 lines

in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

#14 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov
Quoted 3 lines · H. Peter Anvin

Eep. This should be reverted, indeed. This isn't a manifest bug on !Xen but we have gotten requests for WT support which would mean adding in the PAT but again.

Please no, letme fix it. That's what I'm having in mind: don't use pse bit in swap entry but always mark page read from swap as dirty one (it's better than having no tracker at all and will be the case on old machines with 32bit ptes only).

Quite frankly, unless I see a patch later today that is

I'll do my best to fix it, but I've not yet been shown where the problem happens. Neither I'm convinced if it is here at all. I'm asking David for details.

Quoted 9 lines

(a) obvious
(b) explains what is going on
(c) tested

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess

This was not my invention on such encoding, I simply had no choise but follow existing scheme. Still there are cleanup patches in -mm tree which make code a way more readable (even former one, without soft-dirty bit).

Quoted 2 lines

in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

#41 lore raw
In reply to Cyrill Gorcunov
Quoted 22 lines · Cyrill Gorcunov
Quoted 5 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Quoted 7 lines
Quoted 2 lines

orig_pte has pse bit set if page has been soft dirty when it reached swap.

"when it reached swap" to me again implies that it could come from a live page table, with the present bit set. So that explanation attempt of yours confuses me more than it clarifies things for me. (And referring to this bit as PSE bit is

When page swapped out it become non-present in pte entry.

Quoted 7 lines
Quoted 3 lines

sort of wrong here too - there's no PSE bit for 4k PTEs, that bit is the PAT one, and that's what the whole discussion started from.)

And I asked David to point me how it happens, because I don't understand at which point pse bit get analized when page is not present.

As Jan said, we're concerned that the bit was being used on present PTEs and not just non-present ones. From a more careful look at this code this does not appear to be the case.

However, I do find the use of PTE bits in this way somewhat fragile. What other potential corner cases might still remain that will require further games with PTE bits?

FWIW, Xen uses a separate dirty log to track which pages have become dirty since the log was last cleared. Such a dirty log seems more efficient than having scan all the PTEs looking for the soft dirty bits and then having to scan them all again to clear them (particularly if you need multiple passes because the task is still running and continuing to dirty pages).

David

#42 lore raw
In reply to Cyrill Gorcunov
Quoted 22 lines · Cyrill Gorcunov
Quoted 5 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Quoted 7 lines
Quoted 2 lines

orig_pte has pse bit set if page has been soft dirty when it reached swap.

"when it reached swap" to me again implies that it could come from a live page table, with the present bit set. So that explanation attempt of yours confuses me more than it clarifies things for me. (And referring to this bit as PSE bit is

When page swapped out it become non-present in pte entry.

Quoted 7 lines
Quoted 3 lines

sort of wrong here too - there's no PSE bit for 4k PTEs, that bit is the PAT one, and that's what the whole discussion started from.)

And I asked David to point me how it happens, because I don't understand at which point pse bit get analized when page is not present.

As Jan said, we're concerned that the bit was being used on present PTEs and not just non-present ones. From a more careful look at this code this does not appear to be the case.

However, I do find the use of PTE bits in this way somewhat fragile. What other potential corner cases might still remain that will require further games with PTE bits?

FWIW, Xen uses a separate dirty log to track which pages have become dirty since the log was last cleared. Such a dirty log seems more efficient than having scan all the PTEs looking for the soft dirty bits and then having to scan them all again to clear them (particularly if you need multiple passes because the task is still running and continuing to dirty pages).

David

#43 lore raw
Quoted 5 lines · David Vrabel
Quoted 4 lines

And I asked David to point me how it happens, because I don't understand at which point pse bit get analized when page is not present.

Quoted 7 lines

As Jan said, we're concerned that the bit was being used on present PTEs and not just non-present ones. From a more careful look at this code this does not appear to be the case.

However, I do find the use of PTE bits in this way somewhat fragile. What other potential corner cases might still remain that will require further games with PTE bits?

OK, so this is not a bug finally. The problem is that 2 level pte is
quite small and 7th bit is the only one spare I can use for soft dirty tracking when page get swapped out. And swap engine is very depending on pte being non-present, so we are on a safe side.

Quoted 6 lines

FWIW, Xen uses a separate dirty log to track which pages have become dirty since the log was last cleared. Such a dirty log seems more efficient than having scan all the PTEs looking for the soft dirty bits and then having to scan them all again to clear them (particularly if you need multiple passes because the task is still running and continuing to dirty pages).

#44 lore raw
In reply to David Vrabel
Quoted 5 lines · David Vrabel
Quoted 4 lines

And I asked David to point me how it happens, because I don't understand at which point pse bit get analized when page is not present.

Quoted 7 lines

As Jan said, we're concerned that the bit was being used on present PTEs and not just non-present ones. From a more careful look at this code this does not appear to be the case.

However, I do find the use of PTE bits in this way somewhat fragile. What other potential corner cases might still remain that will require further games with PTE bits?

OK, so this is not a bug finally. The problem is that 2 level pte is
quite small and 7th bit is the only one spare I can use for soft dirty tracking when page get swapped out. And swap engine is very depending on pte being non-present, so we are on a safe side.

Quoted 6 lines

FWIW, Xen uses a separate dirty log to track which pages have become dirty since the log was last cleared. Such a dirty log seems more efficient than having scan all the PTEs looking for the soft dirty bits and then having to scan them all again to clear them (particularly if you need multiple passes because the task is still running and continuing to dirty pages).

#83 lore raw
In reply to David Vrabel
Quoted 11 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

Kind of off topic, but do you have a summary of how Xen uses the high PAT bits? I'm the one who wants WT, and if there's already precedent for using the high PAT bits, it'll be helpful.

(Also, Xen is a little weird, and I'd rather not break it again by accident like I did with the vsyscall changes a couple years ago.)

--Andy

#84 lore raw
In reply to David Vrabel
Quoted 11 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

Kind of off topic, but do you have a summary of how Xen uses the high PAT bits? I'm the one who wants WT, and if there's already precedent for using the high PAT bits, it'll be helpful.

(Also, Xen is a little weird, and I'd rather not break it again by accident like I did with the vsyscall changes a couple years ago.)

--Andy

#45 lore raw
In reply to Cyrill Gorcunov
Quoted 9 lines · Cyrill Gorcunov
Quoted 4 lines

However, I do find the use of PTE bits in this way somewhat fragile. What other potential corner cases might still remain that will require further games with PTE bits?

OK, so this is not a bug finally. The problem is that 2 level pte is
quite small and 7th bit is the only one spare I can use for soft dirty tracking when page get swapped out. And swap engine is very depending on pte being non-present, so we are on a safe side.

To make it clear: I'm working on patch that won't use pse bit for dirty page tracking, just give some time to cook it and test.

#46 lore raw
In reply to Cyrill Gorcunov
Quoted 9 lines · Cyrill Gorcunov
Quoted 4 lines

However, I do find the use of PTE bits in this way somewhat fragile. What other potential corner cases might still remain that will require further games with PTE bits?

OK, so this is not a bug finally. The problem is that 2 level pte is
quite small and 7th bit is the only one spare I can use for soft dirty tracking when page get swapped out. And swap engine is very depending on pte being non-present, so we are on a safe side.

To make it clear: I'm working on patch that won't use pse bit for dirty page tracking, just give some time to cook it and test.

#47 lore raw

But is there a manifest bug or not? What is the deal with Xen?

Quoted 17 lines · Cyrill Gorcunov
Quoted 2 lines · Cyrill Gorcunov
Quoted 2 lines

However, I do find the use of PTE bits in this way somewhat

fragile.

Quoted 1 line
Quoted 1 line

What other potential corner cases might still remain that will

require

Quoted 3 lines
Quoted 1 line

further games with PTE bits?

OK, so this is not a bug finally. The problem is that 2 level pte

is

Quoted 1 line

quite small and 7th bit is the only one spare I can use for soft

dirty

Quoted 2 lines

tracking when page get swapped out. And swap engine is very depending on pte being non-present, so we are on a safe side.

To make it clear: I'm working on patch that won't use pse bit for dirty page tracking, just give some time to cook it and test.

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#48 lore raw
In reply to Cyrill Gorcunov

But is there a manifest bug or not? What is the deal with Xen?

Quoted 17 lines · Cyrill Gorcunov
Quoted 2 lines · Cyrill Gorcunov
Quoted 2 lines

However, I do find the use of PTE bits in this way somewhat

fragile.

Quoted 1 line
Quoted 1 line

What other potential corner cases might still remain that will

require

Quoted 3 lines
Quoted 1 line

further games with PTE bits?

OK, so this is not a bug finally. The problem is that 2 level pte

is

Quoted 1 line

quite small and 7th bit is the only one spare I can use for soft

dirty

Quoted 2 lines

tracking when page get swapped out. And swap engine is very depending on pte being non-present, so we are on a safe side.

To make it clear: I'm working on patch that won't use pse bit for dirty page tracking, just give some time to cook it and test.

Signature · 1 line
Sent from my mobile phone. Please excuse brevity and lack of formatting.
#49 lore raw
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

#50 lore raw
In reply to H. Peter Anvin
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

#51 lore raw
Quoted 9 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

Quoted 7 lines

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

I doubt that there are cacheability issues here, since the swap type already overlaps with PWT and PCD.

What happens if the page being swapped is a THP page? (I have no idea how, or even if, this works, but presumably PSE is important.)

--Andy

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC
#52 lore raw
In reply to Cyrill Gorcunov
Quoted 9 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

Quoted 7 lines

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

I doubt that there are cacheability issues here, since the swap type already overlaps with PWT and PCD.

What happens if the page being swapped is a THP page? (I have no idea how, or even if, this works, but presumably PSE is important.)

--Andy

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC
#53 lore raw
Quoted 18 lines · Andy Lutomirski
Quoted 9 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

Quoted 7 lines

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

Quoted 5 lines

I doubt that there are cacheability issues here, since the swap type already overlaps with PWT and PCD.

What happens if the page being swapped is a THP page? (I have no idea how, or even if, this works, but presumably PSE is important.)

add_to_swap -> split_huge_page_to_list, so thp will be splitted.

#54 lore raw
In reply to Andy Lutomirski
Quoted 18 lines · Andy Lutomirski
Quoted 9 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

Quoted 7 lines

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

Quoted 5 lines

I doubt that there are cacheability issues here, since the swap type already overlaps with PWT and PCD.

What happens if the page being swapped is a THP page? (I have no idea how, or even if, this works, but presumably PSE is important.)

add_to_swap -> split_huge_page_to_list, so thp will be splitted.

#55 lore raw
In reply to Andy Lutomirski
Quoted 18 lines · Andy Lutomirski
Quoted 9 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

Quoted 7 lines

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

Quoted 5 lines

I doubt that there are cacheability issues here, since the swap type already overlaps with PWT and PCD.

What happens if the page being swapped is a THP page? (I have no idea how, or even if, this works, but presumably PSE is important.)

Huge-page is splitted into small ones, then the smaller ones get swapped out.

Quoted 2 lines

--Andy

#56 lore raw
In reply to Andy Lutomirski
Quoted 18 lines · Andy Lutomirski
Quoted 9 lines · Cyrill Gorcunov
Quoted 2 lines · H. Peter Anvin

But is there a manifest bug or not? What is the deal with Xen?

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

Quoted 7 lines

- i don't find any code which would test for this bit directly without is_swap_pte call

but the use of paw bit itself is confusing, so I'm working on patch which won't use it. Again, if someone knows where exactly access to pse bit when pte keeps swap entry may happen (for any purpose other than dirty page tracking) please share.

Quoted 5 lines

I doubt that there are cacheability issues here, since the swap type already overlaps with PWT and PCD.

What happens if the page being swapped is a THP page? (I have no idea how, or even if, this works, but presumably PSE is important.)

Huge-page is splitted into small ones, then the smaller ones get swapped out.

Quoted 2 lines

--Andy

#57 lore raw
In reply to Cyrill Gorcunov
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

Linus

#58 lore raw
In reply to Cyrill Gorcunov
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

Linus

#15 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

Quite frankly, unless I see a patch later today that is

(a) obvious
(b) explains what is going on
(c) tested

Quoted 6 lines

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

- only 3level pages scheme uses high 32bits to keep offset of swap entry, x86-64 shifts offset up to _PAGE_BIT_GLOBAL + 1 bit, thus I need some different bit nonunified with anything else for no reason :(

Summarizing all things

- Using PSE bit for swap entries as indicator of soft dirty page is safe because swap entries as saved in pte as non-presen and when #pf happens kernel generates valid pte entry from vma->vm_page_prot

- __swp_entry() helper is clearing PSE bit explicitly so even without softdirty patch it's not saved once page reach swap (with softdirty tracking we simply reuse this bit for own needs).

- Using PSE bit allows to not modify swap encoding on all 3 page schemes (2level, 3level, 4level) because it's a spare bit there not intersected with swap format.

Thus I would *_really_* like to save current scheme. Probably I should add comment into header where _PAGE_SWP_SOFT_DIRTY defined that it's valid only when PRESENT bit clear? Similar to

/* If _PAGE_BIT_PRESENT is clear, we use these: */
/* - if the user mapped it with PROT_NONE; pte_present gives true */
#define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
/* - set: nonlinear file mapping, saved PTE; unset:swap */
#define _PAGE_BIT_FILE _PAGE_BIT_DIRTY

Have I conviced you guys?

The former problem report came from impression that this PSE bit may be touched (set and clean) on present PTE, but it's not the case for pages being swapped.

#16 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

Quite frankly, unless I see a patch later today that is

(a) obvious
(b) explains what is going on
(c) tested

Quoted 6 lines

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

- only 3level pages scheme uses high 32bits to keep offset of swap entry, x86-64 shifts offset up to _PAGE_BIT_GLOBAL + 1 bit, thus I need some different bit nonunified with anything else for no reason :(

Summarizing all things

- Using PSE bit for swap entries as indicator of soft dirty page is safe because swap entries as saved in pte as non-presen and when #pf happens kernel generates valid pte entry from vma->vm_page_prot

- __swp_entry() helper is clearing PSE bit explicitly so even without softdirty patch it's not saved once page reach swap (with softdirty tracking we simply reuse this bit for own needs).

- Using PSE bit allows to not modify swap encoding on all 3 page schemes (2level, 3level, 4level) because it's a spare bit there not intersected with swap format.

Thus I would *_really_* like to save current scheme. Probably I should add comment into header where _PAGE_SWP_SOFT_DIRTY defined that it's valid only when PRESENT bit clear? Similar to

/* If _PAGE_BIT_PRESENT is clear, we use these: */
/* - if the user mapped it with PROT_NONE; pte_present gives true */
#define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
/* - set: nonlinear file mapping, saved PTE; unset:swap */
#define _PAGE_BIT_FILE _PAGE_BIT_DIRTY

Have I conviced you guys?

The former problem report came from impression that this PSE bit may be touched (set and clean) on present PTE, but it's not the case for pages being swapped.

#17 lore raw
Quoted 6 lines · Cyrill Gorcunov

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

Maybe just don't support soft dirty for 2 level page tables?

2 level page tables should be really on the way out anyways, as they have severe limits and do not support NX. With 3 levels there is enough space.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#18 lore raw
In reply to Cyrill Gorcunov
Quoted 6 lines · Cyrill Gorcunov

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

Maybe just don't support soft dirty for 2 level page tables?

2 level page tables should be really on the way out anyways, as they have severe limits and do not support NX. With 3 levels there is enough space.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#59 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

Could this explain what I'm seeing in another thread ?
https://lkml.org/lkml/2013/8/7/27

Dave

#60 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

Could this explain what I'm seeing in another thread ?
https://lkml.org/lkml/2013/8/7/27

Dave

#61 lore raw
Quoted 3 lines · Dave Jones

Could this explain what I'm seeing in another thread ?
https://lkml.org/lkml/2013/8/7/27

Don't think so. This code is merged in -rc6, while your report is saying the kernel version is -rc4 (also this feature requires CONFIG_MEM_SOFT_DIRTY to be turned on, do you have it?).

#19 lore raw
In reply to Andi Kleen
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Cyrill Gorcunov

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

Quoted 8 lines
Quoted 2 lines

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

Maybe just don't support soft dirty for 2 level page tables?

2 level page tables should be really on the way out anyways, as they have severe limits and do not support NX. With 3 levels there is enough space.

Look, good thing is that 7th bit also available on the 4level pages (ie x86-64) without additional code modification, that's why I picked it in first place. I prepared the patch locally which doesn't use pse bit for tracking but it only makes code more complex.

#20 lore raw
In reply to Andi Kleen
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Cyrill Gorcunov

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

Quoted 8 lines
Quoted 2 lines

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

Maybe just don't support soft dirty for 2 level page tables?

2 level page tables should be really on the way out anyways, as they have severe limits and do not support NX. With 3 levels there is enough space.

Look, good thing is that 7th bit also available on the 4level pages (ie x86-64) without additional code modification, that's why I picked it in first place. I prepared the patch locally which doesn't use pse bit for tracking but it only makes code more complex.

#21 lore raw

Hello Cyrill,

Quoted 11 lines · Cyrill Gorcunov
Quoted 10 lines · Andi Kleen
Quoted 6 lines · Cyrill Gorcunov

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

Maybe just don't support soft dirty for 2 level page tables?

Quoted 8 lines
Quoted 3 lines

2 level page tables should be really on the way out anyways, as they have severe limits and do not support NX. With 3 levels there is enough space.

Look, good thing is that 7th bit also available on the 4level pages (ie x86-64) without additional code modification, that's why I picked it in first place. I prepared the patch locally which doesn't use pse bit for tracking but it only makes code more complex.

I'm not sure you read this thread.
http://comments.gmane.org/gmane.linux.kernel.mm/101756
In summary, I'd like to use it to track sub-ranges of some processes.

I already had a time to investigate and it enhanced our workload x2 on ARM so I'd like to expand the concept for more general purpose.

For it, arch-specific stuff would be hurdle for port. So, I support your non-arch-specific solutions. Just FYI. Such future plan shouldn't force you.

Thanks.

Signature · 2 lines
Kind regards,
Minchan Kim
#24 lore raw
In reply to Cyrill Gorcunov

Hello Cyrill,

Quoted 11 lines · Cyrill Gorcunov
Quoted 10 lines · Andi Kleen
Quoted 6 lines · Cyrill Gorcunov

Hi all, I worked on patch which would not touch PSE bit for dirty page tracking and the result is not that good:

- 2level pages now always page dirty if page is swapped in and out, because there is no space left in PTE (other than PSE bit)

Maybe just don't support soft dirty for 2 level page tables?

Quoted 8 lines
Quoted 3 lines

2 level page tables should be really on the way out anyways, as they have severe limits and do not support NX. With 3 levels there is enough space.

Look, good thing is that 7th bit also available on the 4level pages (ie x86-64) without additional code modification, that's why I picked it in first place. I prepared the patch locally which doesn't use pse bit for tracking but it only makes code more complex.

I'm not sure you read this thread.
http://comments.gmane.org/gmane.linux.kernel.mm/101756
In summary, I'd like to use it to track sub-ranges of some processes.

I already had a time to investigate and it enhanced our workload x2 on ARM so I'd like to expand the concept for more general purpose.

For it, arch-specific stuff would be hurdle for port. So, I support your non-arch-specific solutions. Just FYI. Such future plan shouldn't force you.

Thanks.

Signature · 2 lines
Kind regards,
Minchan Kim
#63 lore raw
In reply to Dave Jones
Quoted 16 lines · Dave Jones
Quoted 15 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

Quoted 5 lines
Quoted 2 lines

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

Could this explain what I'm seeing in another thread ?
https://lkml.org/lkml/2013/8/7/27

Was it caught with CONFIG_MEM_SOFT_DIRTY on or off? In the latter case all new bits manipulations are no-op and couldn't cause this.

Quoted 1 line

Dave

Thanks,
Pavel

#64 lore raw
In reply to Dave Jones
Quoted 16 lines · Dave Jones
Quoted 15 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

Quoted 5 lines
Quoted 2 lines

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

Could this explain what I'm seeing in another thread ?
https://lkml.org/lkml/2013/8/7/27

Was it caught with CONFIG_MEM_SOFT_DIRTY on or off? In the latter case all new bits manipulations are no-op and couldn't cause this.

Quoted 1 line

Dave

Thanks,
Pavel

#71 lore raw
In reply to Cyrill Gorcunov
Quoted 5 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 3 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

Quoted 7 lines
Quoted 2 lines

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Fact is that this

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}

has no checking whatsoever that the PTE being modified is a non-present one, not even in any of the debugging modes. It would be a different thing if the above acted on a swp_entry_t.

The fact that there currently may be just a single call site (where the caller guarantees the non-present state) is no guarantee that in the future another one won't appear, and then result in very hard to debug problems.

Jan

#72 lore raw
In reply to Cyrill Gorcunov
Quoted 5 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 3 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

Quoted 7 lines
Quoted 2 lines

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Fact is that this

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}

has no checking whatsoever that the PTE being modified is a non-present one, not even in any of the debugging modes. It would be a different thing if the above acted on a swp_entry_t.

The fact that there currently may be just a single call site (where the caller guarantees the non-present state) is no guarantee that in the future another one won't appear, and then result in very hard to debug problems.

Jan

#73 lore raw
Quoted 20 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 3 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

Quoted 7 lines
Quoted 2 lines

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Fact is that this

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
Quoted 8 lines

has no checking whatsoever that the PTE being modified is a non-present one, not even in any of the debugging modes. It would be a different thing if the above acted on a swp_entry_t.

The fact that there currently may be just a single call site (where the caller guarantees the non-present state) is no guarantee that in the future another one won't appear, and then result in very hard to debug problems.

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
#74 lore raw
In reply to Jan Beulich
Quoted 20 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 3 lines · Jan Beulich
Quoted 2 lines

Only to non-present ptes, as far as I know.

Quoted 7 lines
Quoted 2 lines

That's not really any guarantee. And the accessor functions also don't check that they'd be used on non-present PTEs only.

Wait. This _PAGE_SWP_SOFT_DIRTY bit (which is in real PSE bit) assigned in only one place -- in try_to_unmap_one(). The PTE get non-present then and consists of swap entry format. I don't see any accessor to such entry without testing if it's swap entry or pte-none. What I'm missing?

Fact is that this

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
Quoted 8 lines

has no checking whatsoever that the PTE being modified is a non-present one, not even in any of the debugging modes. It would be a different thing if the above acted on a swp_entry_t.

The fact that there currently may be just a single call site (where the caller guarantees the non-present state) is no guarantee that in the future another one won't appear, and then result in very hard to debug problems.

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
#75 lore raw
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}

Sure, fine with me. Perhaps VM_BUG_ON() or some other similar construct limiting the scope when any extra code gets generated would do too.

But as said, even better would perhaps be to have it act on a swp_entry_t.

Jan

#76 lore raw
In reply to Cyrill Gorcunov
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}

Sure, fine with me. Perhaps VM_BUG_ON() or some other similar construct limiting the scope when any extra code gets generated would do too.

But as said, even better would perhaps be to have it act on a swp_entry_t.

Jan

#65 lore raw
In reply to Linus Torvalds
Quoted 11 lines
Quoted 1 line
Quoted 1 line · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

It was actually me (mis)reading the code - as pointed out to Cyrill already, setting _PAGE_PAT in a pte_t without even a comment saying that this can only ever be done with a non-present entry made me expect problems on Xen, because it's clear that to date bare metal Linux doesn't care about the state of _PAGE_PAT in present entries due to the way the PAT MSR gets set (and hence quite likely no-one would have noticed the supposed problem while testing).

So a comment either alongside the definition of _PAGE_SWP_SOFT_DIRTY or directly in pte_swp_{mk,clear_}soft_dirty() would have been the minimal thing I'd have expected for this sort of re-use of bits. Ideally even a VM_BUG_ON(pte_present()) or similar. And perhaps pte_swp_soft_dirty() should be either looking at the present bit too or similarly asserting that it's clear...

In any event - I'm sorry for the red herring.

Jan

#66 lore raw
In reply to Linus Torvalds
Quoted 11 lines
Quoted 1 line
Quoted 1 line · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

It was actually me (mis)reading the code - as pointed out to Cyrill already, setting _PAGE_PAT in a pte_t without even a comment saying that this can only ever be done with a non-present entry made me expect problems on Xen, because it's clear that to date bare metal Linux doesn't care about the state of _PAGE_PAT in present entries due to the way the PAT MSR gets set (and hence quite likely no-one would have noticed the supposed problem while testing).

So a comment either alongside the definition of _PAGE_SWP_SOFT_DIRTY or directly in pte_swp_{mk,clear_}soft_dirty() would have been the minimal thing I'd have expected for this sort of re-use of bits. Ideally even a VM_BUG_ON(pte_present()) or similar. And perhaps pte_swp_soft_dirty() should be either looking at the present bit too or similarly asserting that it's clear...

In any event - I'm sorry for the red herring.

Jan

#85 lore raw
In reply to Andy Lutomirski
Quoted 14 lines
Quoted 1 line
Quoted 1 line · Andy Lutomirski
Quoted 11 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

Quoted 3 lines

Kind of off topic, but do you have a summary of how Xen uses the high PAT bits? I'm the one who wants WT, and if there's already precedent for using the high PAT bits, it'll be helpful.

Xen's public headers have a comment explaining this, with the main information being this table:

* The PAT MSR is as follow (it is a 64-bit value, each entry is 8 bits):
* PAT4 PAT0
* +---+----+----+----+-----+----+----+
* WC | WC | WB | UC | UC- | WC | WB | <= Linux
* +---+----+----+----+-----+----+----+
* WC | WT | WB | UC | UC- | WT | WB | <= BIOS (default when machine boots)
* +---+----+----+----+-----+----+----+
* WC | WP | WC | UC | UC- | WT | WB | <= Xen
* +---+----+----+----+-----+----+----+

i.e. Xen is retaining the BIOS (and legacy from non-PAT times) meaning of the low four entries, putting WC and WP up into the high half. The fact that the entry 6 is defined to be WC is perhaps a mistake - it should really be considered reserved for an eventual future memory type (just like entry 7). It also seems like entry 6 is documented incorrectly here for Linux and BIOS.

Jan

#86 lore raw
In reply to Andy Lutomirski
Quoted 14 lines
Quoted 1 line
Quoted 1 line · Andy Lutomirski
Quoted 11 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

Quoted 3 lines

Kind of off topic, but do you have a summary of how Xen uses the high PAT bits? I'm the one who wants WT, and if there's already precedent for using the high PAT bits, it'll be helpful.

Xen's public headers have a comment explaining this, with the main information being this table:

* The PAT MSR is as follow (it is a 64-bit value, each entry is 8 bits):
* PAT4 PAT0
* +---+----+----+----+-----+----+----+
* WC | WC | WB | UC | UC- | WC | WB | <= Linux
* +---+----+----+----+-----+----+----+
* WC | WT | WB | UC | UC- | WT | WB | <= BIOS (default when machine boots)
* +---+----+----+----+-----+----+----+
* WC | WP | WC | UC | UC- | WT | WB | <= Xen
* +---+----+----+----+-----+----+----+

i.e. Xen is retaining the BIOS (and legacy from non-PAT times) meaning of the low four entries, putting WC and WP up into the high half. The fact that the entry 6 is defined to be WC is perhaps a mistake - it should really be considered reserved for an eventual future memory type (just like entry 7). It also seems like entry 6 is documented incorrectly here for Linux and BIOS.

Jan

#87 lore raw
Quoted 21 lines · Jan Beulich
Quoted 14 lines
Quoted 1 line
Quoted 1 line · Andy Lutomirski
Quoted 11 lines · David Vrabel

All,

179ef71c (mm: save soft-dirty bits on swapped pages) introduces a new PTE bit on x86 _PTE_SWP_SOFT_DIRTY which has the same value as _PTE_PSE and _PTE_PAT.

With a Xen PV guest, the use of the _PTE_PAT will result in the page having unexpected cachability which will introduce a range of subtle performance and correctness issues. Xen programs the entry 4 in the PAT table with WC so a page that was previously WB will end up as WC.

Quoted 3 lines

Kind of off topic, but do you have a summary of how Xen uses the high PAT bits? I'm the one who wants WT, and if there's already precedent for using the high PAT bits, it'll be helpful.

Xen's public headers have a comment explaining this, with the main information being this table:

Quoted 9 lines

* The PAT MSR is as follow (it is a 64-bit value, each entry is 8 bits):
* PAT4 PAT0
* +---+----+----+----+-----+----+----+
* WC | WC | WB | UC | UC- | WC | WB | <= Linux
* +---+----+----+----+-----+----+----+
* WC | WT | WB | UC | UC- | WT | WB | <= BIOS (default when machine boots)
* +---+----+----+----+-----+----+----+
* WC | WP | WC | UC | UC- | WT | WB | <= Xen
* +---+----+----+----+-----+----+----+

That looks odd and I think the kernel arch/x86/xen/mmu.c has a more correct summary. As much as I remember the specs and the usage in Linux, the upper four slots are mirroring the lower 4 ones. That way PTEs don't need to care to what value the PAT bit is set. The BIOS default would result in the same meaning of PCD and PWT as without PAT. Linux changes it to get WC but otherwise continues to avoid the PAT bit which moves position for large pages.

-Stefan

Quoted 9 lines

i.e. Xen is retaining the BIOS (and legacy from non-PAT times) meaning of the low four entries, putting WC and WP up into the high half. The fact that the entry 6 is defined to be WC is perhaps a mistake - it should really be considered reserved for an eventual future memory type (just like entry 7). It also seems like entry 6 is documented incorrectly here for Linux and BIOS.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#67 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

There are no Xen related bugs in the code, we were misreading it.

It was my call to raise this as a regression without a repro and clearly this was the wrong decision.

However, having looked at the soft dirty implementation and specifically the userspace ABI I think that it is far to closely coupled to the current implementation. I think this will constrain future development of the feature should userspace require a more efficient ABI than scanning all of /proc/<pid>/pagemaps.

Minimal downtime during 'live' checkpointing of a running task needs the checkpointer to find and write out dirty pages faster than the task can dirty them. This seems less likely to be possible if every iteration all PTEs have to be scanned by the checkpointer instead of (e.g.,) accessing a separate list of dirtied pages.

David

#68 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

There are no Xen related bugs in the code, we were misreading it.

It was my call to raise this as a regression without a repro and clearly this was the wrong decision.

However, having looked at the soft dirty implementation and specifically the userspace ABI I think that it is far to closely coupled to the current implementation. I think this will constrain future development of the feature should userspace require a more efficient ABI than scanning all of /proc/<pid>/pagemaps.

Minimal downtime during 'live' checkpointing of a running task needs the checkpointer to find and write out dirty pages faster than the task can dirty them. This seems less likely to be possible if every iteration all PTEs have to be scanned by the checkpointer instead of (e.g.,) accessing a separate list of dirtied pages.

David

#69 lore raw
Quoted 30 lines · David Vrabel
Quoted 10 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

There are no Xen related bugs in the code, we were misreading it.

It was my call to raise this as a regression without a repro and clearly this was the wrong decision.

However, having looked at the soft dirty implementation and specifically the userspace ABI I think that it is far to closely coupled to the current implementation. I think this will constrain future development of the feature should userspace require a more efficient ABI than scanning all of /proc/<pid>/pagemaps.

Quoted 3 lines

Minimal downtime during 'live' checkpointing of a running task needs the checkpointer to find and write out dirty pages faster than the task can dirty them.

Absolutely, but in "find and write" the "write" component is likely to take the majority of time -- we can scan PTEs of a mapping MUCH faster, than transmitting those over even 10Gbit link.

We actually see this IRL -- in CRIU there's an atomic test, that checks mappings get dumped and restored properly. One of sub-tests is one 512Mb mapping. With it total dump time _minus_ memory dump time (which includes not only pagemap file scan, but also files, registers, process tree, sessions, etc.) is fractions of one second, while only the memory dump part's time is several seconds.

That said, the super-fast API for getting "what has changed" is not as tempting to have as faster network/disk.

What is _more_ time consuming in iterative migration in our case is the need to re-scan the whole /proc tree to get which processes had died and appeared, mess with /proc/pid/fd finding out what files were (re-)opened/closed/changed, talking to sock_diag subsystem for sockets information and alike. However, we haven't yet done careful analysis for what the slowest part is, but pagemap scans is definitely not.

Quoted 3 lines

This seems less likely to be possible if every iteration all PTEs have to be scanned by the checkpointer instead of (e.g.,) accessing a separate list of dirtied pages.

But we don't scan all the x64 virtual address space's PTEs, instead we first analyze the /proc/pid/maps and scan only PTEs sitting in private mappings.

Quoted 3 lines

David
.

Thanks,
Pavel

#70 lore raw
In reply to David Vrabel
Quoted 30 lines · David Vrabel
Quoted 10 lines · Linus Torvalds
Quoted 8 lines · Cyrill Gorcunov

I personally don't see bug here because

- this swapped page soft dirty bit is set for non-present entries only, never for present ones, just at moment we form swap pte entry

- i don't find any code which would test for this bit directly without is_swap_pte call

Quoted 7 lines

Ok, having gone through the places that use swp_*soft_dirty(), I have to agree. Afaik, it's only ever used on a swap-entry that has (by definition) the P bit clear. So with or without Xen, I don't see how it can make any difference.

David/Konrad - did you actually see any issues, or was this just from (mis)reading the code?

There are no Xen related bugs in the code, we were misreading it.

It was my call to raise this as a regression without a repro and clearly this was the wrong decision.

However, having looked at the soft dirty implementation and specifically the userspace ABI I think that it is far to closely coupled to the current implementation. I think this will constrain future development of the feature should userspace require a more efficient ABI than scanning all of /proc/<pid>/pagemaps.

Quoted 3 lines

Minimal downtime during 'live' checkpointing of a running task needs the checkpointer to find and write out dirty pages faster than the task can dirty them.

Absolutely, but in "find and write" the "write" component is likely to take the majority of time -- we can scan PTEs of a mapping MUCH faster, than transmitting those over even 10Gbit link.

We actually see this IRL -- in CRIU there's an atomic test, that checks mappings get dumped and restored properly. One of sub-tests is one 512Mb mapping. With it total dump time _minus_ memory dump time (which includes not only pagemap file scan, but also files, registers, process tree, sessions, etc.) is fractions of one second, while only the memory dump part's time is several seconds.

That said, the super-fast API for getting "what has changed" is not as tempting to have as faster network/disk.

What is _more_ time consuming in iterative migration in our case is the need to re-scan the whole /proc tree to get which processes had died and appeared, mess with /proc/pid/fd finding out what files were (re-)opened/closed/changed, talking to sock_diag subsystem for sockets information and alike. However, we haven't yet done careful analysis for what the slowest part is, but pagemap scans is definitely not.

Quoted 3 lines

This seems less likely to be possible if every iteration all PTEs have to be scanned by the checkpointer instead of (e.g.,) accessing a separate list of dirtied pages.

But we don't scan all the x64 virtual address space's PTEs, instead we first analyze the /proc/pid/maps and scan only PTEs sitting in private mappings.

Quoted 3 lines

David
.

Thanks,
Pavel

#77 lore raw
In reply to Jan Beulich
Quoted 9 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
Quoted 3 lines

Sure, fine with me. Perhaps VM_BUG_ON() or some other similar construct limiting the scope when any extra code gets generated would do too.

Sorry for delay, the patch is below.

Quoted 3 lines

But as said, even better would perhaps be to have it act on a swp_entry_t.

swp_entry_t is too small already to keep additional status bit, unfortunately.
---
From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] mm: Make sure _PAGE_SWP_SOFT_DIRTY bit is not set on present pte

_PAGE_SOFT_DIRTY bit should never be set on present pte so add VM_BUG_ON to catch any potential future abuse.

Also add a comment on _PAGE_SWP_SOFT_DIRTY definition explaining scope of its usage.

Cc Andy Lutomirski <[email protected]>
Cc Pavel Emelyanov <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Matt Mackall <[email protected]>
Cc Xiao Guangrong <[email protected]>
Cc Marcelo Tosatti <[email protected]>
Cc KOSAKI Motohiro <[email protected]>
Cc Stephen Rothwell <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc David Vrabel <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc Boris Ostrovsky <[email protected]>
Cc Jan Beulich <[email protected]>
---
 arch/x86/include/asm/pgtable.h       |   34 +++++++++++++++++++---------------
 arch/x86/include/asm/pgtable_types.h |    3 +++
 2 files changed, 22 insertions(+), 15 deletions(-)
arch/x86/include/asm/pgtable.h +19 -15
Index: linux-2.6.git/arch/x86/include/asm/pgtable.h
===================================================================
@@ -314,21 +314,6 @@ static inline pmd_t pmd_mksoft_dirty(pmd
 	return pmd_set_flags(pmd, _PAGE_SOFT_DIRTY);
 }
 
-static inline pte_t pte_swp_mksoft_dirty(pte_t pte)
-{
-	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
-}
-
-static inline int pte_swp_soft_dirty(pte_t pte)
-{
-	return pte_flags(pte) & _PAGE_SWP_SOFT_DIRTY;
-}
-
-static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
-{
-	return pte_clear_flags(pte, _PAGE_SWP_SOFT_DIRTY);
-}
-
 static inline pte_t pte_file_clear_soft_dirty(pte_t pte)
 {
 	return pte_clear_flags(pte, _PAGE_SOFT_DIRTY);
@@ -445,6 +430,7 @@ pte_t *populate_extra_pte(unsigned long
 
 #ifndef __ASSEMBLY__
 #include <linux/mm_types.h>
+#include <linux/mmdebug.h>
 #include <linux/log2.h>
 
 static inline int pte_none(pte_t pte)
@@ -863,6 +849,24 @@ static inline void update_mmu_cache_pmd(
 {
 }
 
+static inline pte_t pte_swp_mksoft_dirty(pte_t pte)
+{
+	VM_BUG_ON(pte_present(pte));
+	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
+}
+
+static inline int pte_swp_soft_dirty(pte_t pte)
+{
+	VM_BUG_ON(pte_present(pte));
+	return pte_flags(pte) & _PAGE_SWP_SOFT_DIRTY;
+}
+
+static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
+{
+	VM_BUG_ON(pte_present(pte));
+	return pte_clear_flags(pte, _PAGE_SWP_SOFT_DIRTY);
+}
+
 #include <asm-generic/pgtable.h>
 #endif	/* __ASSEMBLY__ */
 
arch/x86/include/asm/pgtable_types.h +3 -0
Index: linux-2.6.git/arch/x86/include/asm/pgtable_types.h
===================================================================
@@ -75,6 +75,9 @@
  * with swap entry format. On x86 bits 6 and 7 are *not* involved
  * into swap entry computation, but bit 6 is used for nonlinear
  * file mapping, so we borrow bit 7 for soft dirty tracking.
+ *
+ * Please note that this bit must be treated as swap dirty page
+ * mark if and only if the PTE has present bit clear!
  */
 #ifdef CONFIG_MEM_SOFT_DIRTY
 #define _PAGE_SWP_SOFT_DIRTY	_PAGE_PSE
#78 lore raw
In reply to Jan Beulich
Quoted 9 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
Quoted 3 lines

Sure, fine with me. Perhaps VM_BUG_ON() or some other similar construct limiting the scope when any extra code gets generated would do too.

Sorry for delay, the patch is below.

Quoted 3 lines

But as said, even better would perhaps be to have it act on a swp_entry_t.

swp_entry_t is too small already to keep additional status bit, unfortunately.
---
From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] mm: Make sure _PAGE_SWP_SOFT_DIRTY bit is not set on present pte

_PAGE_SOFT_DIRTY bit should never be set on present pte so add VM_BUG_ON to catch any potential future abuse.

Also add a comment on _PAGE_SWP_SOFT_DIRTY definition explaining scope of its usage.

Cc Andy Lutomirski <[email protected]>
Cc Pavel Emelyanov <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Matt Mackall <[email protected]>
Cc Xiao Guangrong <[email protected]>
Cc Marcelo Tosatti <[email protected]>
Cc KOSAKI Motohiro <[email protected]>
Cc Stephen Rothwell <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc David Vrabel <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Konrad Rzeszutek Wilk <[email protected]>
Cc Boris Ostrovsky <[email protected]>
Cc Jan Beulich <[email protected]>
---
 arch/x86/include/asm/pgtable.h       |   34 +++++++++++++++++++---------------
 arch/x86/include/asm/pgtable_types.h |    3 +++
 2 files changed, 22 insertions(+), 15 deletions(-)
arch/x86/include/asm/pgtable.h +19 -15
Index: linux-2.6.git/arch/x86/include/asm/pgtable.h
===================================================================
@@ -314,21 +314,6 @@ static inline pmd_t pmd_mksoft_dirty(pmd
 	return pmd_set_flags(pmd, _PAGE_SOFT_DIRTY);
 }
 
-static inline pte_t pte_swp_mksoft_dirty(pte_t pte)
-{
-	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
-}
-
-static inline int pte_swp_soft_dirty(pte_t pte)
-{
-	return pte_flags(pte) & _PAGE_SWP_SOFT_DIRTY;
-}
-
-static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
-{
-	return pte_clear_flags(pte, _PAGE_SWP_SOFT_DIRTY);
-}
-
 static inline pte_t pte_file_clear_soft_dirty(pte_t pte)
 {
 	return pte_clear_flags(pte, _PAGE_SOFT_DIRTY);
@@ -445,6 +430,7 @@ pte_t *populate_extra_pte(unsigned long
 
 #ifndef __ASSEMBLY__
 #include <linux/mm_types.h>
+#include <linux/mmdebug.h>
 #include <linux/log2.h>
 
 static inline int pte_none(pte_t pte)
@@ -863,6 +849,24 @@ static inline void update_mmu_cache_pmd(
 {
 }
 
+static inline pte_t pte_swp_mksoft_dirty(pte_t pte)
+{
+	VM_BUG_ON(pte_present(pte));
+	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
+}
+
+static inline int pte_swp_soft_dirty(pte_t pte)
+{
+	VM_BUG_ON(pte_present(pte));
+	return pte_flags(pte) & _PAGE_SWP_SOFT_DIRTY;
+}
+
+static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
+{
+	VM_BUG_ON(pte_present(pte));
+	return pte_clear_flags(pte, _PAGE_SWP_SOFT_DIRTY);
+}
+
 #include <asm-generic/pgtable.h>
 #endif	/* __ASSEMBLY__ */
 
arch/x86/include/asm/pgtable_types.h +3 -0
Index: linux-2.6.git/arch/x86/include/asm/pgtable_types.h
===================================================================
@@ -75,6 +75,9 @@
  * with swap entry format. On x86 bits 6 and 7 are *not* involved
  * into swap entry computation, but bit 6 is used for nonlinear
  * file mapping, so we borrow bit 7 for soft dirty tracking.
+ *
+ * Please note that this bit must be treated as swap dirty page
+ * mark if and only if the PTE has present bit clear!
  */
 #ifdef CONFIG_MEM_SOFT_DIRTY
 #define _PAGE_SWP_SOFT_DIRTY	_PAGE_PSE
Acked-by
#79 lore raw
Quoted 28 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 9 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
Quoted 3 lines

Sure, fine with me. Perhaps VM_BUG_ON() or some other similar construct limiting the scope when any extra code gets generated would do too.

Sorry for delay, the patch is below.

Quoted 3 lines

But as said, even better would perhaps be to have it act on a swp_entry_t.

swp_entry_t is too small already to keep additional status bit, unfortunately.
---
From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] mm: Make sure _PAGE_SWP_SOFT_DIRTY bit is not set on present pte

Quoted 7 lines

_PAGE_SOFT_DIRTY bit should never be set on present pte so add VM_BUG_ON to catch any potential future abuse.

Also add a comment on _PAGE_SWP_SOFT_DIRTY definition explaining scope of its usage.

Thanks, Jan

Acked-by
#80 lore raw
In reply to Cyrill Gorcunov
Quoted 28 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov
Quoted 9 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · Cyrill Gorcunov

Ok, how about this?

static inline pte_t pte_swp_mksoft_dirty(pte_t pte)

{
	BUG_ON(pte_present(pte));
	return pte_set_flags(pte, _PAGE_SWP_SOFT_DIRTY);
}
Quoted 3 lines

Sure, fine with me. Perhaps VM_BUG_ON() or some other similar construct limiting the scope when any extra code gets generated would do too.

Sorry for delay, the patch is below.

Quoted 3 lines

But as said, even better would perhaps be to have it act on a swp_entry_t.

swp_entry_t is too small already to keep additional status bit, unfortunately.
---
From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] mm: Make sure _PAGE_SWP_SOFT_DIRTY bit is not set on present pte

Quoted 7 lines

_PAGE_SOFT_DIRTY bit should never be set on present pte so add VM_BUG_ON to catch any potential future abuse.

Also add a comment on _PAGE_SWP_SOFT_DIRTY definition explaining scope of its usage.

Thanks, Jan

#81 lore raw
Quoted 7 lines · Jan Beulich
Quoted 7 lines

From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] mm: Make sure _PAGE_SWP_SOFT_DIRTY bit is not set on present pte

_PAGE_SOFT_DIRTY bit should never be set on present pte so add VM_BUG_ON to catch any potential future abuse.

Quoted 6 lines
Quoted 4 lines

Also add a comment on _PAGE_SWP_SOFT_DIRTY definition explaining scope of its usage.

#82 lore raw
In reply to Jan Beulich
Quoted 7 lines · Jan Beulich
Quoted 7 lines

From: Cyrill Gorcunov <[email protected]>
Subject: [PATCH] mm: Make sure _PAGE_SWP_SOFT_DIRTY bit is not set on present pte

_PAGE_SOFT_DIRTY bit should never be set on present pte so add VM_BUG_ON to catch any potential future abuse.

Quoted 6 lines
Quoted 4 lines

Also add a comment on _PAGE_SWP_SOFT_DIRTY definition explaining scope of its usage.

#22 lore raw
In reply to Minchan Kim
Quoted 10 lines · Minchan Kim
Quoted 5 lines

Look, good thing is that 7th bit also available on the 4level pages (ie x86-64) without additional code modification, that's why I picked it in first place. I prepared the patch locally which doesn't use pse bit for tracking but it only makes code more complex.

I'm not sure you read this thread.
http://comments.gmane.org/gmane.linux.kernel.mm/101756
In summary, I'd like to use it to track sub-ranges of some processes.

Quoted 6 lines

I already had a time to investigate and it enhanced our workload x2 on ARM so I'd like to expand the concept for more general purpose.

For it, arch-specific stuff would be hurdle for port. So, I support your non-arch-specific solutions. Just FYI. Such future plan shouldn't force you.

Hi Minchan, thanks for info, I'll read the thread (sorry for delay, managed to miss your email).

#23 lore raw
In reply to Minchan Kim
Quoted 10 lines · Minchan Kim
Quoted 5 lines

Look, good thing is that 7th bit also available on the 4level pages (ie x86-64) without additional code modification, that's why I picked it in first place. I prepared the patch locally which doesn't use pse bit for tracking but it only makes code more complex.

I'm not sure you read this thread.
http://comments.gmane.org/gmane.linux.kernel.mm/101756
In summary, I'd like to use it to track sub-ranges of some processes.

Quoted 6 lines

I already had a time to investigate and it enhanced our workload x2 on ARM so I'd like to expand the concept for more general purpose.

For it, arch-specific stuff would be hurdle for port. So, I support your non-arch-specific solutions. Just FYI. Such future plan shouldn't force you.

Hi Minchan, thanks for info, I'll read the thread (sorry for delay, managed to miss your email).

#25 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

Anything that makes me try to scavenge a new PTE bits makes me scream :-) Dunno if I'll manage to support this on power.

Also, it sort-of duplicates what KVM does for dirty tracking (for migration, framebuffer updates, etc...). I wonder if KVM could consider switching to this scheme, but then we end up with a user "break KVM" file in /proc since the user can clear the refs.

I'd have been happier if the whole thing had essentially used a parallel set of dirty tracking bitmaps (hooked up with the VMAs maybe). Add the overhead there for as many "clients" as you want who will use the facility and leave the PTE mostly alone basically. (I suppose we still need to play PTE tricks to differenciate soft dirty RO vs. COW RO on anonymous memory ?)

Cheers,
Ben.

#26 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

I will be reverting the whole soft-dirty mess. I thought the bit-mapping games it played were already too complicated (the patch to pgtable-2level.h in commit 41bb3476b361 just makes me want to barf and came in very late, so I'm not positive about the whole soft-dirty mess in the first place). I really am not at all inclined to want to play games in this area any more. It's too damn late in the release window.

Anything that makes me try to scavenge a new PTE bits makes me scream :-) Dunno if I'll manage to support this on power.

Also, it sort-of duplicates what KVM does for dirty tracking (for migration, framebuffer updates, etc...). I wonder if KVM could consider switching to this scheme, but then we end up with a user "break KVM" file in /proc since the user can clear the refs.

I'd have been happier if the whole thing had essentially used a parallel set of dirty tracking bitmaps (hooked up with the VMAs maybe). Add the overhead there for as many "clients" as you want who will use the facility and leave the PTE mostly alone basically. (I suppose we still need to play PTE tricks to differenciate soft dirty RO vs. COW RO on anonymous memory ?)

Cheers,
Ben.