New: xfs transaction overruns on 4.14.67
https://bugzilla.kernel.org/show_bug.cgi?id=202077
Bug ID: 202077
Summary: xfs transaction overruns on 4.14.67
Product: File System
Version: 2.5
Kernel Version: 4.14.67
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: NoCreated attachment 280149
--> https://bugzilla.kernel.org/attachment.cgi?id=280149&action=edit
xfs transaction overrun #1
We've encountered two recent examples of xfs transaction overruns on production systems running 4.14.67 kernels. Both systems in this case are running docker with dozens of overlay mounts, using this xfs fs as both upper and lower. In both cases the filesystem was able to successfully recover when the filesystem was unmounted and remounted again.
It looks like there has been a good bit of work in 4.16+ addressing similar issues but none of it has made it back into the 4.14 LTS. Any chance that any of the attached debug output points to anything specific that might be a candidate for backport?
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 202077] xfs transaction overruns on 4.14.67
https://bugzilla.kernel.org/show_bug.cgi?id=202077
--- Comment #1 from Thomas Walker ([email protected]) --- Created attachment 280151
--> https://bugzilla.kernel.org/attachment.cgi?id=280151&action=edit
xfs transaction overrun #2
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
Quoted 5 lines · On Wed, Dec 26, 2018 at 05:19:12PM +0000, [email protected]…
We've encountered two recent examples of xfs transaction overruns on production systems running 4.14.67 kernels. Both systems in this case are running docker with dozens of overlay mounts, using this xfs fs as both upper and lower. In both cases the filesystem was able to successfully recover when the filesystem was unmounted and remounted again.
Inboth cases, it looks like there were two free space manipulations in a single transaction, likely first modifying the free list (pattern is EFD, XAGF, ABTB, ABTC, then AGFL) followed by freeing the actual extent (more ABTB, ABTC buffers).
Quoted 1 line
It looks like there has been a good bit of work in 4.16+
The first fixes went into 4.18 with the deferred AGFL free operations. Those were the commits associated with the patchset titled "[PATCH v2 0/6] xfs: defer agfl block frees".
There were more fixes in 4.19 to always defer the AGFL free for all operations. This was a much larger and more significant change, and can be found from the series titled "[PATCH 00/24] xfs: broad enablement of deferred agfl frees".
Quoted 3 lines
addressing similar issues but none of it has made it back into the 4.14 LTS. Any chance that any of the attached debug output points to anything specific that might be a candidate for backport?
Backporting the first series might be sufficient to avoid your problem (both are from the inode inactivation path) but it is no guarantee. I also have no idea what dependencies that patchset has on the rest of the code (e.g. is there enough deferred op infrastructure in place in 4.14?), and seeing as it touches core allocation algorithms it would require a substantial amount of QA before release....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
[Bug 202077] xfs transaction log reservation overruns on 4.14.67
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=202077
--- Comment #2 from Dave Chinner ([email protected]) ---
Quoted 8 lines · On Wed, Dec 26, 2018 at 05:19:12PM +0000, [email protected]…
We've encountered two recent examples of xfs transaction overruns on production
systems running 4.14.67 kernels. Both systems in this case are running docker
with dozens of overlay mounts, using this xfs fs as both upper and lower. In both cases the filesystem was able to successfully recover when the filesystem
was unmounted and remounted again.
Inboth cases, it looks like there were two free space manipulations in a single transaction, likely first modifying the free list (pattern is EFD, XAGF, ABTB, ABTC, then AGFL) followed by freeing the actual extent (more ABTB, ABTC buffers).
Quoted 1 line
It looks like there has been a good bit of work in 4.16+
The first fixes went into 4.18 with the deferred AGFL free operations. Those were the commits associated with the patchset titled "[PATCH v2 0/6] xfs: defer agfl block frees".
There were more fixes in 4.19 to always defer the AGFL free for all operations. This was a much larger and more significant change, and can be found from the series titled "[PATCH 00/24] xfs: broad enablement of deferred agfl frees".
Quoted 3 lines
addressing similar issues but none of it has made it back into the 4.14 LTS. Any chance that any of the attached debug output points to anything specific that might be a candidate for backport?
Backporting the first series might be sufficient to avoid your problem (both are from the inode inactivation path) but it is no guarantee. I also have no idea what dependencies that patchset has on the rest of the code (e.g. is there enough deferred op infrastructure in place in 4.14?), and seeing as it touches core allocation algorithms it would require a substantial amount of QA before release....
Cheers,
Dave.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.
[Bug 202077] xfs transaction log reservation overruns on 4.14.67
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=202077
--- Comment #3 from Thomas Walker ([email protected]) --- Thanks for the response. That first patchset does appear to apply cleanly (with a little fuzz) to 4.14 but, as you say, I don't know offhand how mature the code it depends upon is in 4.14 and without a reliable reproducer it will be hard to say whether it even addresses my issue. I'll keep seeing if I can reproduce the problem more consistently and see...
While I'm running 4.19 on a few test systems, I've been taking a wait-and-see approach towards broader usage given the number of regressions that have cropped up (and been fixed) thus far. Good to know that this is likely addressed there though.
Thanks,
Tom.
Signature · 2 lines
You are receiving this mail because: You are watching the assignee of the bug.