swapoff() runs forever
Hi!
I'm observing a strange issue (at least on UML) on recent Linux kernels. If swap is being used the swapoff() system call never terminates. To be precise "while ((i = find_next_to_unuse(si, i)) != 0)" in try_to_unuse() never terminates.
The affected machine has 256MiB ram and 256MiB swap.
If an application uses more than 256MiB memory swap is being used. But after the application terminates the free command still reports that a few MiB are on my swap device and swappoff never terminates.
Here some numbers:
root@linux:~# free
total used free shared buffers cached
Mem: 255472 13520 241952 0 312 7080
-/+ buffers/cache: 6128 249344
Swap: 262140 17104 245036
root@linux:~# cat /proc/meminfo
MemTotal: 255472 kB
MemFree: 241952 kB
Buffers: 312 kB
Cached: 7080 kB
SwapCached: 0 kB
Active: 3596 kB
Inactive: 6076 kB
Active(anon): 1512 kB
Inactive(anon): 848 kB
Active(file): 2084 kB
Inactive(file): 5228 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 262140 kB
SwapFree: 245036 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 2296 kB
Mapped: 1824 kB
Shmem: 80 kB
Slab: 2452 kB
SReclaimable: 1116 kB
SUnreclaim: 1336 kB
KernelStack: 192 kB
PageTables: 556 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 389876 kB
Committed_AS: 238412 kB
VmallocTotal: 3788784 kB
VmallocUsed: 68 kB
VmallocChunk: 3788716 kBWhat could cause this issue?
I'm not sure whether this is UML specific or not.
Maybe only UML is able to trigger the issue...
Thanks,
//richard
Quoted 11 lines · Richard Weinberger
Hi!
I'm observing a strange issue (at least on UML) on recent Linux kernels. If swap is being used the
swapoff()system call never terminates. To be precise "while ((i =find_next_to_unuse(si, i)) != 0)" intry_to_unuse()never terminates.The affected machine has 256MiB ram and 256MiB swap.
If an application uses more than 256MiB memory swap is being used. But after the application terminates the free command still reports that a few MiB are on my swap device and swappoff never terminates.
After last tmpfs changes swapoff can take minutes. Or this time it really never terminates?
Quoted 42 lines
Here some numbers:
root@linux:~# free total used free shared buffers cached Mem: 255472 13520 241952 0 312 7080 -/+ buffers/cache: 6128 249344 Swap: 262140 17104 245036 root@linux:~# cat /proc/meminfo MemTotal: 255472 kB MemFree: 241952 kB Buffers: 312 kB Cached: 7080 kB SwapCached: 0 kB Active: 3596 kB Inactive: 6076 kB Active(anon): 1512 kB Inactive(anon): 848 kB Active(file): 2084 kB Inactive(file): 5228 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 262140 kB SwapFree: 245036 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 2296 kB Mapped: 1824 kB Shmem: 80 kB Slab: 2452 kB SReclaimable: 1116 kB SUnreclaim: 1336 kB KernelStack: 192 kB PageTables: 556 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 389876 kB Committed_AS: 238412 kB VmallocTotal: 3788784 kB VmallocUsed: 68 kB VmallocChunk: 3788716 kB
Quoted 6 lines
What could cause this issue?
I'm not sure whether this is UML specific or not.
Maybe only UML is able to trigger the issue...Thanks,
//richard
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Am 09.04.2012 07:35, schrieb Konstantin Khlebnikov:
Quoted 8 lines
Quoted 7 lines · Richard Weinberger
Hi!
I'm observing a strange issue (at least on UML) on recent Linux kernels. If swap is being used the
swapoff()system call never terminates. To be precise "while ((i =find_next_to_unuse(si, i)) != 0)" intry_to_unuse()never terminates.
Quoted 7 lines
Quoted 4 lines
The affected machine has 256MiB ram and 256MiB swap.
If an application uses more than 256MiB memory swap is being used. But after the application terminates the free command still reports that a few MiB are on my swap device and swappoff never terminates.After last tmpfs changes swapoff can take minutes. Or this time it really never terminates?
I've never waited forever. ;-)
Once I've waited for >30 minutes.
I don't think that it's related to tmpfs because it happens
also while shutting down the system after all filesystems have been unmounted.
Thanks,
//richard
Quoted 14 lines · Richard Weinberger
Am 09.04.2012 07:35, schrieb Konstantin Khlebnikov:
Quoted 13 lines
Quoted 11 lines · Richard Weinberger
Hi!
I'm observing a strange issue (at least on UML) on recent Linux kernels. If swap is being used the
swapoff()system call never terminates. To be precise "while ((i =find_next_to_unuse(si, i)) != 0)" intry_to_unuse()never terminates.The affected machine has 256MiB ram and 256MiB swap.
If an application uses more than 256MiB memory swap is being used. But after the application terminates the free command still reports that a few MiB are on my swap device and swappoff never terminates.
Quoted 4 lines
Quoted 2 lines
After last tmpfs changes swapoff can take minutes. Or this time it really never terminates?
I've never waited forever. ;-)
Your lack of dedication is disappointing.
Quoted 4 lines
Once I've waited for >30 minutes.
I don't think that it's related to tmpfs because it happens
also while shutting down the system after all filesystems have been unmounted.
Like you I'd assume that it is really was going to be forever, rather than swapoff just being characteristically slow:
a few MiB left on swap shouldn't take long to get off.
I've not seen any such issue in recent months (or years), but I've not been using UML either. The most likely cause that springs to mind would be corruption of the vmalloc'ed swap map: that would be very likely to cause such a hang.
You say "recent Linux kernels": I wonder what "recent" means. Is this something you can reproduce quickly and reliably enough to do a bisection upon?
Thanks,
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Am 09.04.2012 20:40, schrieb Hugh Dickins:
Quoted 27 lines · Richard Weinberger
Quoted 14 lines
Am 09.04.2012 07:35, schrieb Konstantin Khlebnikov:
Quoted 13 lines
Quoted 11 lines · Richard Weinberger
Hi!
I'm observing a strange issue (at least on UML) on recent Linux kernels. If swap is being used the
swapoff()system call never terminates. To be precise "while ((i =find_next_to_unuse(si, i)) != 0)" intry_to_unuse()never terminates.The affected machine has 256MiB ram and 256MiB swap.
If an application uses more than 256MiB memory swap is being used. But after the application terminates the free command still reports that a few MiB are on my swap device and swappoff never terminates.Quoted 4 lines
Quoted 2 lines
After last tmpfs changes swapoff can take minutes. Or this time it really never terminates?
I've never waited forever. ;-)
Your lack of dedication is disappointing.
Quoted 4 lines
Once I've waited for>30 minutes.
I don't think that it's related to tmpfs because it happens
also while shutting down the system after all filesystems have been unmounted.
Quoted 8 lines
Like you I'd assume that it is really was going to be forever, rather than swapoff just being characteristically slow:
a few MiB left on swap shouldn't take long to get off.I've not seen any such issue in recent months (or years), but I've not been using UML either. The most likely cause that springs to mind would be corruption of the vmalloc'ed swap map: that would be very likely to cause such a hang.
Okay, I'll dig into this.
Quoted 4 lines
You say "recent Linux kernels": I wonder what "recent" means. Is this something you can reproduce quickly and reliably enough to do a bisection upon?
It happens quite reliably on 3.2 and 3.3.
On 3.1 and 3.0 sometimes.
I've already wasted half a day with bisecting it.
Thanks,
//richard
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Am 09.04.2012 20:40, schrieb Hugh Dickins:
Quoted 4 lines
I've not seen any such issue in recent months (or years), but I've not been using UML either. The most likely cause that springs to mind would be corruption of the vmalloc'ed swap map: that would be very likely to cause such a hang.
It does not look like a swap map corruption.
If I restart most user space processes swapoff() terminates fine. Maybe it is a refcounting problem?
Quoted 4 lines
You say "recent Linux kernels": I wonder what "recent" means. Is this something you can reproduce quickly and reliably enough to do a bisection upon?
I can reproduce the issue on any UML kernel.
The oldest I've tested was 2.6.20.
Therefore, bug was not introduced by me. B-)
Thanks,
//richard
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Richard Weinberger
Am 09.04.2012 20:40, schrieb Hugh Dickins:
Quoted 4 lines
I've not seen any such issue in recent months (or years), but I've not been using UML either. The most likely cause that springs to mind would be corruption of the vmalloc'ed swap map: that would be very likely to cause such a hang.
It does not look like a swap map corruption.
If I restart most user space processesswapoff()terminates fine.
Right, thanks, that's very useful info.
Quoted 1 line
Maybe it is a refcounting problem?
You may prove to be correct; but since killing and restarting processes fixes it up without (I presume) issuing warnings, it doesn't sound like a refcounting problem to me.
Quoted 9 lines
Quoted 5 lines
You say "recent Linux kernels": I wonder what "recent" means. Is this something you can reproduce quickly and reliably enough to do a bisection upon?
I can reproduce the issue on any UML kernel. The oldest I've tested was 2.6.20. Therefore, bug was not introduced by me. B-)
More useful info, thank you.
I think I've spotted two problems in the UML swp_entry_t handling; but checking if I'm right, and if they're relevant, and how to fix them, I'll leave to you - it's years since I tried UML and I remember 0.
One, likely to be your problem. Take a look at unuse_pte_range() in mm/swapfile.c, where it searches the page table for the swp_pte it's trying to "unuse". And take a look at set_pte() in arch/um/include/asm/pgtable.h, which appears to add a mysterious _PAGE_NEWPAGE bit into the page table entry. And UML doesn't provide an alternative to generic pte_same() in include/asm-genric/pgtable.h.
My guess is that the _NEWPAGE bit prevents swapoff from matching pte against swap entry in all or some cases (I didn't look to see if _NEWPAGE is sometimes cleared later).
Probably a good fix to try would be providing a UML pte_same() which takes that into account; but I don't know what conditionals it should contain, and whether it would become too inefficient. Or, if _NEWPAGE is always set in a swap pte, then swp_entry_to_pte() needs to set it.
(A word of warning if you're unfamiliar with swap entries: there's the kernel's internal representation swp_entry_t, which has offset in the low-order and type in the high-order, for efficient use with radix_tree
- see include/linux/swapops.h; and then there's the arch-dependent representation as a page table entry, which rearranges the bits so as not to be confused with a good present page table entry, and traditionally has type on the lower side of offset.)
The other thing I noticed first, probably not relevant to the bug you're seeing since I think you'd have mentioned if you had two swapfiles; but the two or more swapfile case looks very broken to me. _PAGE_PROTNONE is 0x010 but __swp_type(x) is (((x).val >> 4) & 0x3f): unless I'm confused, a swap entry of type 1 will look just like a PROT_NONE pte.
Or maybe that's resolved by the _PAGE_NEWPAGE and _PAGE_NEWPROT bits, I didn't spend time working out what they're up to.
include/linux/swap.h does not allow MAX_SWAPFILES to exceed 32, so you can easily change __swp_type(x) to use 5 and 0x1f instead (with 5 instead of 4 in __swp_entry too of course). Though it doesn't cause error, I wonder where the 11 in __swp_offset and __swp_entry comes from: I think you can support larger swap by making it 10.
Hugh
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>