truncate head of file?
Was thinking it would be very handy to have a truncate() variant that deletes pages from the head of a file. This could be leveraged to make logfiles easier to maintain, as an example. Anyone else interested, think this would be nice to have?
(Note - not the same as just punching holes in the beginning of the file - we want the beginning of the file to advance as well, past the deleted pages.)
Signature · 4 lines
-- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Quoted 3 lines · Howard Chu
Was thinking it would be very handy to have a
truncate()variant that deletes pages from the head of a file. This could be leveraged to make logfiles easier to maintain, as an example. Anyone else interested, think this would be nice to have?(Note - not the same as just punching holes in the beginning of the file - we want the beginning of the file to advance as well, past the deleted pages.)
What is a "page" of a file?
Thanks,
Signature · 1 line
~Randy
Quoted 5 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?
Quoted 7 lines
Was thinking it would be very handy to have a
truncate()variant that deletes pages from the head of a file. This could be leveraged to make logfiles easier to maintain, as an example. Anyone else interested, think this would be nice to have?(Note - not the same as just punching holes in the beginning of the file - we want the beginning of the file to advance as well, past the deleted pages.)
I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
Thanks!
-Lukas
Quoted 12 lines · Lukáš Czerner
Quoted 10 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?Was thinking it would be very handy to have a
truncate()variant that deletes pages from the head of a file. This could be leveraged to make logfiles easier to maintain, as an example. Anyone else interested, think this would be nice to have?
Quoted 6 lines
Quoted 2 lines
(Note - not the same as just punching holes in the beginning of the file - we want the beginning of the file to advance as well, past the deleted pages.)
I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
For example - we have a logfile (opened O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file the read() would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
Freeing the space would be simplest if we just deallocate X pages from the file, and then the beginning of the file becomes the beginning of the first remaining page of the file.
Signature · 8 lines
-- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 39 lines · Howard Chu
Date: Wed, 20 Aug 2014 00:00:56 -0700
From: Howard Chu <[email protected]>
To: Lukáš Czerner <[email protected]>
Cc: linux-fsdevel <[email protected]>
Subject: Re: truncate head of file?Quoted 15 lines · Lukáš Czerner
Quoted 13 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?Was thinking it would be very handy to have a
truncate()variant that deletes
pages from the head of a file. This could be leveraged to make logfiles easier
to maintain, as an example. Anyone else interested, think this would be nice
to have?Quoted 8 lines
Quoted 4 lines
(Note - not the same as just punching holes in the beginning of the file - we
want the beginning of the file to advance as well, past the deleted pages.)I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
For example - we have a logfile (opened
O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file theread()would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
Quoted 3 lines
Freeing the space would be simplest if we just deallocate X pages from the file, and then the beginning of the file becomes the beginning of the first remaining page of the file.
Ok, now I understand. It is exactly what FALLOC_FL_COLLAPSE_RANGE is for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61
However it will not work with O_APPEND nor does any other fallocate flag except pure fallocate, so not even punch hole would have worked.
-Lukas
Quoted 41 lines · Lukáš Czerner
Quoted 39 lines · Howard Chu
Date: Wed, 20 Aug 2014 00:00:56 -0700
From: Howard Chu <[email protected]>
To: Lukáš Czerner <[email protected]>
Cc: linux-fsdevel <[email protected]>
Subject: Re: truncate head of file?Quoted 15 lines · Lukáš Czerner
Quoted 13 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?Was thinking it would be very handy to have a
truncate()variant that deletes
pages from the head of a file. This could be leveraged to make logfiles easier
to maintain, as an example. Anyone else interested, think this would be nice
to have?Quoted 8 lines
Quoted 4 lines
(Note - not the same as just punching holes in the beginning of the file - we
want the beginning of the file to advance as well, past the deleted pages.)I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
For example - we have a logfile (opened
O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file theread()would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
Quoted 6 lines
Quoted 3 lines
Freeing the space would be simplest if we just deallocate X pages from the file, and then the beginning of the file becomes the beginning of the first remaining page of the file.
Ok, now I understand. It is exactly what
FALLOC_FL_COLLAPSE_RANGEis for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61
Ah, thanks for the tip.
Quoted 3 lines
However it will not work with
O_APPENDnor does any other fallocate flag except pure fallocate, so not even punch hole would have worked.
Hm, that's a little bit inconvenient. What if the process calling fallocate() is different from the one appending to the log (i.e., using two separate file descriptors)? Also, when a range is collapsed, are all processes with descriptors opened on the file updated - are their seek positions correctly decremented? I dug up the patches and didn't see any code to handle this but may have missed it.
Signature · 8 lines
-- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 67 lines · Howard Chu
Quoted 59 lines · Lukáš Czerner
Quoted 49 lines · Howard Chu
Date: Wed, 20 Aug 2014 00:00:56 -0700
From: Howard Chu <[email protected]>
To: Lukáš Czerner <[email protected]>
Cc: linux-fsdevel <[email protected]>
Subject: Re: truncate head of file?Quoted 23 lines · Lukáš Czerner
Quoted 13 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?Was thinking it would be very handy to have a
truncate()variant that deletes
pages from the head of a file. This could be leveraged to make logfiles easier
to maintain, as an example. Anyone else interested, think this would be nice
to have?Quoted 5 lines
(Note - not the same as just punching holes in the beginning of the file -
we
want the beginning of the file to advance as well, past the deleted pages.)Quoted 3 lines
I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
For example - we have a logfile (opened
O_APPEND) that grows continuously. We
want to delete some old log info from the head of the file. We could use "hole
punching" to cause a specific range of data to be freed, but that just leaves
a sparse file. If we were to cat this file theread()would have to advance
thru all of that empty space before arriving at actual log data. We want both
the data to be freed and for the logical beginning of the file to be moved
forward, to match the location of where the remaining data begins.Quoted 5 lines
Freeing the space would be simplest if we just deallocate X pages from the
file, and then the beginning of the file becomes the beginning of the first
remaining page of the file.Quoted 2 lines
Ok, now I understand. It is exactly what
FALLOC_FL_COLLAPSE_RANGEis for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61Ah, thanks for the tip.
Quoted 6 lines
Quoted 3 lines
However it will not work with
O_APPENDnor does any other fallocate flag except pure fallocate, so not even punch hole would have worked.Hm, that's a little bit inconvenient. What if the process calling
There seems to be some confusion between per fd flag O_APPEND and inode flag S_APPEND.
Collapse range can work well with O_APPEND flag but it will not work if S_APPEND is set.
Quoted 5 lines
fallocate()is different from the one appending to the log (i.e., using two separate file descriptors)? Also, when a range is collapsed, are all processes with descriptors opened on the file updated - are their seek positions correctly decremented? I dug up the patches and didn't see any code to handle this but may have missed it.
You are right. Currently seek positions are not updated. But this will not be of any problem (at least not on XFS) for your use case if file is open with O_APPEND.
Refer to this discussion:
https://lkml.org/lkml/2014/2/26/701
Quoted 11 lines
--
-- Howard ChuCTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 73 lines · Ashish Sangwan
Quoted 72 lines · Howard Chu
Quoted 59 lines · Lukáš Czerner
Quoted 49 lines · Howard Chu
Date: Wed, 20 Aug 2014 00:00:56 -0700
From: Howard Chu <[email protected]>
To: Lukáš Czerner <[email protected]>
Cc: linux-fsdevel <[email protected]>
Subject: Re: truncate head of file?Quoted 23 lines · Lukáš Czerner
Quoted 13 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?Was thinking it would be very handy to have a
truncate()variant that deletes
pages from the head of a file. This could be leveraged to make logfiles easier
to maintain, as an example. Anyone else interested, think this would be nice
to have?Quoted 5 lines
(Note - not the same as just punching holes in the beginning of the file -
we
want the beginning of the file to advance as well, past the deleted pages.)Quoted 3 lines
I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
For example - we have a logfile (opened
O_APPEND) that grows continuously. We
want to delete some old log info from the head of the file. We could use "hole
punching" to cause a specific range of data to be freed, but that just leaves
a sparse file. If we were to cat this file theread()would have to advance
thru all of that empty space before arriving at actual log data. We want both
the data to be freed and for the logical beginning of the file to be moved
forward, to match the location of where the remaining data begins.Quoted 5 lines
Freeing the space would be simplest if we just deallocate X pages from the
file, and then the beginning of the file becomes the beginning of the first
remaining page of the file.Quoted 2 lines
Ok, now I understand. It is exactly what
FALLOC_FL_COLLAPSE_RANGEis for. It has been merged into mainline with commit v3.14-rc1-1-g00f5e61Ah, thanks for the tip.
Quoted 3 lines
However it will not work with
O_APPENDnor does any other fallocate flag except pure fallocate, so not even punch hole would have worked.
Quoted 5 lines
Quoted 1 line
Hm, that's a little bit inconvenient. What if the process calling
There seems to be some confusion between per fd flag
O_APPENDand inode flagS_APPEND.
Collapse range can work well withO_APPENDflag but it will not work ifS_APPENDis set.
Ah ok, that's much better.
Quoted 10 lines
Quoted 5 lines
fallocate()is different from the one appending to the log (i.e., using two separate file descriptors)? Also, when a range is collapsed, are all processes with descriptors opened on the file updated - are their seek positions correctly decremented? I dug up the patches and didn't see any code to handle this but may have missed it.You are right. Currently seek positions are not updated. But this will not be of any problem (at least not on XFS) for your use case if file is open with
O_APPEND.
Refer to this discussion:
https://lkml.org/lkml/2014/2/26/701
Thanks, looks good.
I find it somewhat surprising that such a useful feature was implemented for the benefit of video editing, when it has massive applicability in databases (particularly for simplified truncation of write-ahead logs, as well as for append-only database designs). This is such an obviously beneficial feature but I haven't seen it talked up very much. Of course, being so specific to very recent Linux releases will certainly impede its adoption.
Signature · 8 lines
-- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Howard Chu
For example - we have a logfile (opened
O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file theread()would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
It would also be possible to use fallocate(FALLOC_FL_PUNCH_HOLE) to punch out the processed records, and seek(SEEK_DATA) to find the first record that has not been processed. That would avoid problems with the file size changing and O_APPEND.
Cheers, Andreas
Quoted 8 lines · Andreas Dilger
Quoted 1 line · Howard Chu
For example - we have a logfile (opened
O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file theread()would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.It would also be possible to use
fallocate(FALLOC_FL_PUNCH_HOLE)to punch out the processed records, andseek(SEEK_DATA)to find the first record that has not been processed. That would avoid problems with the file size changing andO_APPEND.
That would be OK, except that we want the file size to decrease. Otherwise the logical size grows without bound and eventually we'd have to recreate the file anyway.
Signature · 4 lines
-- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Quoted 16 lines · Howard Chu
Quoted 15 lines · Lukáš Czerner
Quoted 12 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?Was thinking it would be very handy to have a
truncate()variant that deletes pages from the head of a file. This could be leveraged to make logfiles easier to maintain, as an example. Anyone else interested, think this would be nice to have?(Note - not the same as just punching holes in the beginning of the file - we want the beginning of the file to advance as well, past the deleted pages.)
Quoted 8 lines
Quoted 3 lines
I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
For example - we have a logfile (opened
O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file theread()would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.Freeing the space would be simplest if we just deallocate X pages from the file, and then the beginning of the file becomes the beginning of the first remaining page of the file.
Hi Howard,
I am still interested in what you mean by a "page" of a file? Is it fixed size? What determines page size?
Thanks,
Signature · 5 lines
~Randy -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 23 lines · Randy Dunlap
Quoted 22 lines · Howard Chu
Quoted 12 lines · Lukáš Czerner
Quoted 10 lines · Howard Chu
Date: Tue, 19 Aug 2014 22:45:16 -0700
From: Howard Chu <[email protected]>
To: linux-fsdevel <[email protected]>
Subject: truncate head of file?Was thinking it would be very handy to have a
truncate()variant that deletes pages from the head of a file. This could be leveraged to make logfiles easier to maintain, as an example. Anyone else interested, think this would be nice to have?Quoted 6 lines
Quoted 2 lines
(Note - not the same as just punching holes in the beginning of the file - we want the beginning of the file to advance as well, past the deleted pages.)
I am not really sure I understand the behaviour you'd like to see. Can you please explain the behaviour including more concrete use case ?
For example - we have a logfile (opened
O_APPEND) that grows continuously. We want to delete some old log info from the head of the file. We could use "hole punching" to cause a specific range of data to be freed, but that just leaves a sparse file. If we were to cat this file theread()would have to advance thru all of that empty space before arriving at actual log data. We want both the data to be freed and for the logical beginning of the file to be moved forward, to match the location of where the remaining data begins.
Quoted 7 lines
Quoted 2 lines
Freeing the space would be simplest if we just deallocate X pages from the file, and then the beginning of the file becomes the beginning of the first remaining page of the file.
Hi Howard,
I am still interested in what you mean by a "page" of a file? Is it fixed size? What determines page size?
I meant a typical page as used by the memory manager. For the most part that means e.g. 4KB since "large" page sizes are not actually pageable.
Signature · 8 lines
-- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html